domino logo
4.1
  • Tech Ecosystem
  • Get Started
  • Domino Cloud
  • Collaborate
  • Projects
  • Work with Data
  • Workspaces
  • Environments
  • Executions
  • Deploy Models and Apps
  • Organizations
  • Security and Credentials
  • Notifications
  • Search Your Domino Deployment
  • Domino CLI
  • Troubleshooting
  • Get Help
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
User Guide
>
Workspaces
>
Clusters
>
Spark on Domino
>
On-Demand Spark
>
Manage Dependencies

Manage Dependencies

In a shared Spark cluster, it can be challenging for teams to manage their dependencies (for example, Python packages or JARs). Installing every dependency that a Spark application may need before it runs and dealing with version conflicts can be complex and time-consuming.

Domino allows you to easily package and manage dependencies as part of your Spark-enabled compute environments. This approach creates the flexibility to manage dependencies for individual projects or workloads without having to deal with the complexity of a shared cluster.

To add a new dependency, add the appropriate statements in the Docker Instructions section of the relevant Spark and execution compute environments.

For example to add numpy, include the following.

### Optionally specify version if desired
RUN pip install numpy
Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.