Before you can start using on-demand Spark clusters on Domino you need to ensure that this functionality is enabled and properly configured on your deployment.
Domino Administrators must enable on-demand Spark functionality by setting the ShortLived.SparkClustersEnabled
feature flag to true
.
Set ShortLived.UseNewWorkspaceChrome
to true.
This enables the new Domino workspace
experience which allows for easy access to the Spark Web UI.
By default, Domino does not come with a Spark compatible Compute Environment that can be used for the components of the cluster. Without at least one such environment available, you will not be able to create a cluster.
When using on-demand Spark in Domino you will have two separate environments - one for the Spark cluster (base or worker environment) and one for the workspace/job execution (compute environment).
To create a new base Spark cluster environment, you will follow the
general
Environment Management
with the following environment_attributes
.
-
Base image
Select Custom Image and specify an image URI that points to a deployable Spark image.