Jobs provide a structured way to execute code files using the required hardware, environment, data, security, and other features of Domino. Jobs are executed as part of a Project and live inside the Project scope.
Key information about each Job is automatically captured to ensure the reproducibility and auditability of work. See Monitor Jobs to learn more about saved Job output and how to access Jobs.
Under the hood, Jobs are a type of execution where an executor machine is assigned to execute a specified command in its OS shell. When you start a Job, Domino launches a new environment for your code on the target executor. You can start multiple concurrent Jobs. Each gets its own container environment, so you can try multiple parameters and techniques in parallel. You can schedule recurring Jobs and set up notifications, including customized reports.
Depending on the specific needs and nature of your work, you may want to consider using Workspaces instead of Jobs:
When to use a Job
-
Model training & large-scale computations — If you need to train models on a large dataset or run any sort of long-running, computationally intensive tasks.
-
Batch processing — If you have tasks you want to run in the background and/or in parallel.
-
Reproducibility — If strict reproducibility is important, Jobs provide a guarantee that the exact same execution Environment can be re-created.
-
Automation — If the task needs to be run on a regular schedule or is part of a retraining or automated workflow.
When to use a Workspace
-
Exploratory data analysis (EDA) — If you are in the initial stages of a data science Project and need to explore and analyze your data interactively.
-
Rapid model iterations — If you want to iterate rapidly on model development, hyperparameter tuning, or feature engineering and need rapid feedback with the ability to make instant adjustments.
-
Code development & debugging — If you need an Environment to write, test, and modify code in real-time and see results immediately.