Datasets best practices¶
This article describes how to use Domino Datasets to solve problems, improve collaboration, and open new workflow possibilities in Domino.
You can use Datasets to…¶
When you start a Run or launch a Workspace, Domino copies your project files to a Domino execution. When working with large volumes of data, this presents three potential problems:
- The number of files that can be stored in Domino project files may exceed the configurable limit. By default, the limit is 10,000 files.
- There is a limit to the size of any individual file that can be transferred to and from your Domino project files. By default, this limit is 8GB.
- The time required to transfer data to and from the Domino executions is proportional to the size of the data. It can take a long time if the size of the data is very large, leading to long startup and shutdown times for Workspaces, Jobs, Apps, and Launchers.
You can solve these problems with Domino Datasets:
- There is no limit to the number of files that can be stored in a Domino Dataset.
- There is no limit to the size of any individual file stored in a Domino Dataset.
- Domino Datasets are directly attached to executions as networked filesystems, removing the need to transfer their contents when executions start or complete.
For reproducibility of training experiments it is often helpful to version the contents of a Domino Dataset so that you can go back the specific version used in the past.
There are two ways you can accomplish this with Domino:
- Explicitly create versions of a Domino dataset by creating a Snapshot
- Use a naming convention and a folder hierarchy to organize data your way in the read/write portions of a dataset.
If you use the Domino CLI to work with projects to your local machine, you may find that storing large data files slows down your download and sync operations, and fills up a lot of your local disk storage. You can prevent this by storing data in a Domino Dataset, and reserving your project files for the scripts and documents you want to work with locally.
Follow these steps to simplify your local workflow:
Once the files have been written to the Dataset, you can remove them from your project files.
Update your code to reference your data files in their new location, at:
When everything is working smoothly, you can delete any copies of the project from your local machine that have the large data files in them.
If you have data in an external source that you want to periodically fetch and load into Domino, you can do so with scheduled Jobs set up to write to Datasets.
Suppose you have data stored in an external data source that is periodically updated. If you wanted to fetch the latest state of that file once per week and load it into a Domino Dataset, you could use the following process to set up a scheduled Run.
- Create a Dataset to store the data from the external source.
- Write a script that fetches the data and writes it to the Dataset.
- Create a scheduled job to run your script with the new Dataset configuration.
Below is a detailed example showing how to fetch a large, dynamic data file from a private S3 bucket with a scheduled Run once per week.
First, create a Dataset
to hold the file. This example shows the Dataset being named
For this example, assume the S3 bucket is named my_bucket and the file
you want is named
some_data.csv. In that case, you can set up your
script like this:
import boto3 import io # create new S3 client client = boto3.client('s3') # download some_data.csv from my_bucket and write to latest-S3 output mount file = client.download_file('my_bucket', 'some_data.csv', '/domino/datasets/fetched-from-s3/some_data.csv')
The last step is to set up a scheduled Job that executes this script once per week with the correct Dataset configuration.