About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Create custom Job types

Learn about Domino’s natively supported Job types and languages, as well as how to create custom Job types to support your specific needs.

Domino has native support for running a wide variety of different types of Jobs, both in a single container and using on-demand, auto-scaling clusters. Additionally, Domino lets you customize the type of programming languages and applications that can be used in Domino Jobs to achieve nearly any type of computational workload. This flexibility lets you do things like large-scale data preparation, distributed GPU deep learning model training, and more.

Natively supported Job types

The following lists the supported file types and the commands that Domino executes when it runs the file type:

LanguageFiletype/KeywordInvocation

Python

py

python -u

R

r

R --no-save --no-environ --no-site-file --no-restore --max-ppsize=100000

Spark

--spark

python /spark-submit-wrapper.py

Markdown

rmd/rhtml

R -e require('knitr'); knit('file_name')

Bash

sh

bash file_name

Matlab

m/mlx/mexa64

matlab –sd $(dirname file_name) -batch

Perl

pl

perl file_name

Julia

jl

julia file_name

Python notebook

ipynb

ipython nbconvert -to notebook --execute file_name --output file_name

Fsharp

fsx

fsharpi file_name

Sas

sas

sas file_name

Stata

do

stata -b do

Custom Job types

Domino supports executing nearly any type of user application code to achieve a wide variety of computational scenarios by editing an Environment definition to support your needs.

Note
 This feature is disabled by default. To use this feature, your administrator must set the Configuration records key com.cerebro.domino.executableTarget.enabled to true.

What is a file handler?

When you run a Job, Domino automatically handles the execution of the file. For natively supported file types, Domino already has commands it runs based on the file extension. To execute a file extension that isn’t natively supported, or to run a different command for a file extension that is in the list, you can use a file handler.

A file handler is a bash script with the name of the file extension (without the .sh extension), located in /opt/domino/handlers/{extension}.

File handler examples

The file handler lives in the Environment’s file system, so the execution of the Job depends on the Environment in which you run the Job.

  • The file handler /opt/domino/handlers/jpeg targets files with the .jpeg extension.

  • The file handler /opt/domino/handlers/py overrides the default behavior for .py files.

Tip

Always include instructions to install the necessary software and dependencies to execute the file type. For example, if you create a file handler to add support for Node.js, include a line to install nodejs:

RUN apt-get -y install nodejs

Add a file handler through the Domino UI

To add a file handler through the Domino UI, edit an Environment definition.

  1. Click Environments in the Domino sidebar.

  2. In the Dockerfile instructions field, create a directory for the custom file handler:

    RUN mkdir -p /opt/domino/bin/file-handlers

  3. Add the commands you want to execute when you invoke the file. Save the custom commands to a file with the same name as the file extension you want to target:

    RUN echo "#!/bin/bash" >> /opt/domino/bin/file-handlers/<target-file-extension>
RUN echo "<Line 1 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
RUN echo "<Line 2 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
...

Add a file handler through the terminal

To add a file handler through the terminal, you must build a Docker image and upload it to an external repository. You can use the Docker image to create an Environment that contains your file handler.

  1. Create a directory for the custom file handler:

    mkdir -p /opt/domino/bin/file-handlers

  2. Add the commands you want to execute when you invoke the file. Save the custom commands to a file with the same name as the file extension you want to target:

    RUN echo "#!/bin/bash" >> /opt/domino/bin/file-handlers/<target-file-extension>
RUN echo "<Line 1 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
RUN echo "<Line 2 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
...

  3. Save a copy of this file to a Docker image:

    FROM {DOCKER_BASE_IMAGE}
RUN mkdir -p /opt/domino/bin/file-handlers
COPY <target-file-extension> /opt/domino/bin/file-handlers/

  4. Build the Docker image and upload it to an external repository associated with your Project.

    docker build -t "repository/domino-env-with-executable-target" .
docker push "repository/domino-env-with-executable-target"

Share a custom file handler

When you add custom file handlers to an Environment, the file handlers are associated with that Environment’s Docker image. Each Environment has its own Docker image. You can only have one Docker image per Environment.

If you must use your custom file handler in another Environment, you have several alternatives:

  1. If the file handler is in the Dockerfile instructions field, you can copy and paste the handler’s commands into the same field in a different Environment.

  2. If the file handler is in a Docker image in an external repository:

    1. Use the Docker image as a base to build Docker images for any Environment that needs your custom file handler.

    2. Upload the file handler itself to an external repository. You can reference this file from multiple Docker images.

Next step

Learn about all the ways you can launch a Job.