domino logo
Tech Ecosystem
Get started with Python
Step 0: Orient yourself to DominoStep 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Get started with R
Step 0: Orient yourself to Domino (R Tutorial)Step 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Get Started with MATLAB
Step 1: Orient yourself to DominoStep 2: Create a Domino ProjectStep 3: Configure Your Domino ProjectStep 4: Start a MATLAB WorkspaceStep 5: Fetch and Save Your DataStep 6: Develop Your ModelStep 7: Clean Up Your Workspace
Step 8: Deploy Your Model
Scheduled JobsLaunchers
Step 9: Working with Domino Datasets
Domino Reference
Notifications
On-Demand Open MPI
Configure MPI PrerequisitesFile Sync MPI ClustersValidate MPI VersionWork with your ClusterManage Dependencies
Projects
Projects OverviewProjects PortfolioReference ProjectsProject Goals in Domino 4+
Git Integration
Git Repositories in DominoGit-based Projects with CodeSyncWorking from a Commit ID in Git
Jira Integration in DominoUpload Files to Domino using your BrowserFork and Merge ProjectsSearchSharing and CollaborationCommentsDomino Service FilesystemComparing File RevisionsRevert Projects and Files
Advanced Project Settings
Project DependenciesProject TagsRename a ProjectSet up your Project to Ignore FilesUpload files larger than 550MBExporting Files as a Python or R PackageTransfer Project Ownership
Domino Runs
JobsDiagnostic Statistics with dominostats.jsonNotificationsResultsRun Comparison
Advanced Options for Domino Runs
Run StatesDomino Environment VariablesEnvironment Variables for Secure Credential StorageUse Apache Airflow with Domino
Scheduled Jobs
Domino Workspaces
WorkspacesUse Git in Your WorkspaceRecreate A Workspace From A Previous CommitUse Visual Studio Code in Domino WorkspacesPersist RStudio PreferencesAccess Multiple Hosted Applications in one Workspace Session
Spark on Domino
On-Demand Spark
On-Demand Spark OverviewValidated Spark VersionConfigure PrerequisitesWork with your ClusterManage DependenciesWork with Data
External Hadoop and Spark
Hadoop and Spark OverviewConnecting to a Cloudera CDH5 cluster from DominoConnecting to a Hortonworks cluster from DominoConnect to a MapR cluster from DominoConnect to an Amazon EMR cluster from DominoRunning Local Spark on a Domino ExecutorUsing PySpark in Jupyter WorkspacesKerberos Authentication
On-Demand Ray
On-Demand Ray OverviewValidated Ray VersionConfigure PrerequisitesWork with your ClusterManage DependenciesWork with Data
On-Demand Dask
On-Demand Dask OverviewValidated Dask VersionConfigure PrerequisitesWork with Your ClusterManage DependenciesWork with Data
Customize the Domino Software Environment
Environment ManagementDomino Standard EnvironmentsInstall Packages and DependenciesAdd Workspace IDEsAdding Jupyter KernelsAutomatic Adaptation of Custom Images
Partner Environments for Domino
Use MATLAB as a WorkspaceUse Stata as a WorkspaceUse SAS as a Workspace
Advanced Options for Domino Software Environment
Publish in Domino with Custom ImagesInstall Custom Packages in Domino with Git IntegrationAdd Custom DNS Servers to Your Domino EnvironmentConfigure a Compute Environment to User Private Cran/Conda/PyPi MirrorsUse TensorBoard in Jupyter Workspaces
Publish your Work
Publish a Model API
Model Publishing OverviewModel Invocation SettingsModel Access and CollaborationModel Deployment ConfigurationPromote Projects to ProductionExport Model ImageExport to NVIDIA Fleet Command
Publish a Web Application
App Publishing OverviewGet Started with DashGet Started with ShinyGet Started with FlaskContent Security Policies for Web Apps
Advanced Web Application Settings in Domino
App Scaling and PerformanceHost HTML Pages from DominoHow to Get the Domino Username of an App Viewer
Launchers
Launchers OverviewAdvanced Launcher Editor
Assets Portfolio Overview
Model Monitoring and Remediation
Monitor WorkflowsData Drift and Quality Monitoring
Set up Monitoring for Model APIs
Set up Prediction CaptureSet up Drift DetectionSet up Model Quality MonitoringSet up NotificationsSet Scheduled ChecksSet up Cohort Analysis
Set up Model Monitor
Connect a Data SourceRegister a ModelSet up Drift DetectionSet up Model Quality MonitoringSet up Cohort AnalysisSet up NotificationsSet Scheduled Checks
Use Monitoring
Access the Monitor DashboardAnalyze Data DriftAnalyze Model QualityExclude Features from Scheduled Checks
Remediation
Cohort Analysis
Review the Cohort Analysis
Remediate a Model API
Monitor Settings
API TokenHealth DashboardNotification ChannelsTest Defaults
Monitoring Config JSON
Supported Binning Methods
Model Monitoring APIsTroubleshoot the Model Monitor
Connect to your Data
Data in Domino
Datasets OverviewProject FilesDatasets Best Practices
Connect to Data Sources
External Data VolumesDomino Data Sources
Connect to External Data
Connect Domino to DataRobotConnect to Azure Data Lake StorageConnect to BigQuery from DominoConnect to Google Cloud StorageConnect to IBM DB2 from DominoConnect to IBM Netezza from DominoConnect to Impala from DominoConnect to MSSQL from DominoConnect to MySQL from DominoConnect to Okera from DominoConnect to Oracle Database from DominoConnect to PostgreSQL from DominoConnect to Redshift from DominoConnect to S3 from DominoConnect to SnowflakeConnect to Teradata from Domino
Work with Data Best Practices
Work with Big Data in DominoWork with Lots of FilesMove Data Over a Network
Advanced User Configuration Settings
User API KeysDomino TokenOrganizations Overview
Use the Domino Command Line Interface (CLI)
Install the Domino Command Line (CLI)Domino CLI ReferenceDownload Files with the CLIForce-Restore a Local ProjectMove a Project Between Domino DeploymentsUse the Domino CLI Behind a Proxy
Browser Support
Get Help with Domino
Additional ResourcesGet Domino VersionContact Domino Technical SupportSupport Bundles
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
User Guide
>
Domino Reference
>
Connect to your Data
>
Data in Domino
>
Datasets Best Practices

Datasets Best Practices

This topic describes how to use Domino Datasets to solve problems, improve collaboration, and open new workflow possibilities in Domino.

The following sections describe how you can use datasets.

Store more files, bigger files, and access them faster

When you start a Run or launch a Workspace, Domino copies your project files to a Domino execution. When working with large volumes of data, this presents three potential problems:

  1. The number of files that can be stored in Domino project files may exceed the configurable limit. By default, the limit is 10,000 files.

  2. There is a limit to the size of any individual file that can be transferred to and from your Domino project files. By default, this limit is 8GB.

  3. The time required to transfer data to and from the Domino executions is proportional to the size of the data. It can take a long time if the size of the data is very large, leading to long startup and shutdown times for Workspaces, Jobs, Apps, and Launchers.

You can solve these problems with Domino Datasets:

  1. There is no limit to the number of files that can be stored in a Domino Dataset.

  2. There is no limit to the size of any individual file stored in a Domino Dataset.

  3. Domino Datasets are directly attached to executions as networked filesystems, removing the need to transfer their contents when executions start or complete.

Select a versioning scheme that fits your needs

For reproducibility of training experiments it is often helpful to version the contents of a Domino Dataset so that you can go back the specific version used in the past.

There are two ways you can accomplish this with Domino:

  • Explicitly create versions of a Domino dataset by creating a Snapshot

  • Use a naming convention and a folder hierarchy to organize data your way in the read/write portions of a dataset.

Build multiple curated collections of shareable data in your project

If you use project imports and exports to share data with other members of your team, the consumers of your project will receive the entire contents of your project files in their Runs and Workspaces. That works well if your project is small, simple, and narrowly scoped.

However, for large projects that produce many data artifacts, you may want to expose them to your consumers in smaller, curated subsets. You can do this with Domino Datasets.

Consider the following project.

Screen Shot 2019 01 30 at 12.30.10 PM

This project has a small folder full of code, plus nine folders full of various kinds of output data. Each data folder is larger than 10GB, and the whole project is 100GB. It would be impractical to ask your data consumers to import this project, but you also don’t want to separate the data from the code that produced it by moving the data to a different project.

The solution is to organize the data into Datasets, with one Dataset for each type of data your consumers are interested in. In this example, suppose you have two colleagues who want to consume your data. One of them is only interested in the data from the experiment1 folder, and the other is only interested in the data from experiment9.

You can follow the instructions on the Dataset overview to create and write to two Datasets with scripts like the following, where it’s assumed you have named the Datasets experiment1-data and experiment9-data.

experiment1-populate-dataset.sh
cp -R $DOMINO_WORKING_DIR/experiment1/. /domino/datasets/experiment1-data/
experiment9-populate-dataset.sh
cp -R $DOMINO_WORKING_DIR/experiment9/. /domino/datasets/experiment9-data/

Your consumers can then mount only the Datasets they are interested in.

If you are working with data at this scale, you should write it to Datasets whenever you produce it, instead of storing it in your project files.

Simplify working with Domino locally

If you use the Domino CLI to work with projects to your local machine, you may find that storing large data files slows down your download and sync operations, and fills up a lot of your local disk storage. You can prevent this by storing data in a Domino Dataset, and reserving your project files for the scripts and documents you want to work with locally.

Follow these steps to simplify your local workflow:

  1. Create a Dataset in your project, and write your large data files to it.

  2. After the files have been written to the Dataset, you can remove them from your project files.

  3. Fetch a fresh, slimmed-down copy of your project.

  4. Update your code to reference your data files in their new location, at:

    /domino/datasets/local/<dataset-name>/

  5. When everything is working smoothly, you can delete any copies of the project from your local machine that have the large data files in them.

Automatically pipe data from external sources into Domino

If you have data in an external source that you want to periodically fetch and load into Domino, you can do so with scheduled Jobs set up to write to Datasets.

Suppose you have data stored in an external data source that is periodically updated. If you wanted to fetch the latest state of that file once per week and load it into a Domino Dataset, you could use the following process to set up a scheduled Run.

  1. Create a Dataset to store the data from the external source.

  2. Write a script that fetches the data and writes it to the Dataset.

  3. Create a scheduled job to run your script with the new Dataset configuration.

The following is a detailed example showing how to fetch a large, dynamic data file from a private S3 bucket with a scheduled Run once per week.

First, create a Dataset to hold the file. This example shows the Dataset being named fetched-from-s3.

datasets best practices from s3

For this example, assume the S3 bucket is named my_bucket and the file you want is named some_data.csv. In that case, you can set up your script like this:

fetch-data.py

import boto3
import io

# create new S3 client
client = boto3.client('s3')

# download some_data.csv from my_bucket and write to latest-S3 output mount
file = client.download_file('my_bucket',
                            'some_data.csv',
                            '/domino/datasets/fetched-from-s3/some_data.csv')

The last step is to set up a scheduled Job that executes this script once per week with the correct Dataset configuration.

Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.