About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Domino 5.1.0 (February 2022)

See also the fleetcommand-agent Release Notes.

Validated Frameworks

The following versions have been validated with Domino 5.1. Other versions might also be compatible but are not guaranteed.

New Features

Support for external data volumes (EDVs)

You can now use generic external data volumes (EDVs) with Domino. The new Generic volume type lets you attach your own volumes to Domino that you can deploy and maintain in your environment. See External Data Volumes for more information.

Support for more data sources

This release includes new data connectors to support the following data sources:

See Connecting to external data sources for details.

Export to NVIDIA Fleet Command

A new public API endpoint /:modelId/:modelVersionId/exportImageForNvidia is used to export a Domino Model to be deployed by NVIDIA Fleetcommand. See Export to NVIDIA Fleet Command for details.

Automatic compatibility for custom images

Domino 5.1 can automatically adapt your custom Docker images for compatibility with the Domino platform, making it easier to import a custom environment that you can use with Domino jobs and workspaces. For example, you can use this feature to use external catalogs of pre-built images like the GPU-optimized NVIDIA GPU Cloud (NGC) containers. See Automatic Adaptation of Custom Images in Domino Environments for instructions.

Notifications

Admins can now create and manage custom notifications to send to Domino users. For example, you can use this feature to inform users about planned maintenance, remind them to save their work, or send onboarding messages to a small group of new hires. This feature is enabled by default with the ShortLived.EnableUserNotifications feature flag.

notifications

Admins can create and manage notifications at Admin > Notifications. Users can view their notifications by clicking Notifications in the navigation pane.

Improvements

  • Custom certificates

    You can configure Domino to connect to services that use custom certificates that are external to the Domino cluster. In addition to public services like AWS S3, you might want to use private services in your security domain that are secured with custom certificates or a custom certificate authority. See Custom Certificates.

  • Support for legacy DAD environments

    Starting with this release, Domino Analytics Distribution (DAD) environments from Domino 3.x are supported. This allows you to upgrade to Domino 5.1 while continuing to use legacy versions of the DAD environment image. Domino still recommends that you upgrade to the latest version of the Domino Standard Environment (DSE) to get the best out of Domino’s newest capabilities.

  • Full-name matching in collaborator search

    When you search for collaborators, Domino now matches first and last names in addition to usernames.

  • System administrators must now have the manage-users role for the realm-client to activate and deactivate users in Domino. Admins without this permission will receive a 403 error indicating that they are not authorized to perform these actions. See Set permissions to activate or deactivate users for details.

  • Preferences about job run notifications now include an All failed runs option so that collaborators who have dependencies on each other’s jobs can be notified when a job fails.

  • The dispatcher.url setting is removed from application.conf and Settings.scala. This setting pertained to the now-deprecated Dispatcher dashboard.

  • This release includes several UI improvements. For example, you can click a table heading to sort by ascending, sort by descending, or un-sort.

  • Due to a change in administrator permissions, you must be the project owner to add, modify, or delete environment variables. See Set up project environment variables.

API changes

  • Starting with Domino 5.1, liveness probes for nucleus pods are based on the /liveness endpoint, and readiness probes are based on the /readiness endpoint. This is a simpler implementation for these probes than the /version endpoint and reduces uncertainty during troubleshooting.

  • A new public API endpoint /:modelId/:modelVersionId/exportImageForNvidia is used to export a Domino Model to be deployed by NVIDIA Fleetcommand. See Export to NVIDIA Fleet Command for details.

  • A set of APIs for Notifications is available to perform both user and administrator functions. For users, the APIs can check for unread notifications, show the current user’s notifications, and mark notifications as read. For administrators, the APIs can create notifications, list notifications, update existing notifications, and delete notifications.

  • The request body for the "create scheduled job" endpoint has changed. POST requests sent to /v4/projects/<projectId>/scheduledjobs must now contain an environmentRevisionSpec in the request body. Its value can be ActiveRevision, LatestRevision, or an environmentRevisionSpec ID.

Bug Fixes

  • Fixed an issue that can occur in Domino 5.0.0 where going to the Admin > Executions page might produce a 500 error if a hardware tier that was previously used for a workspace is then set to be cluster-specific by a Domino admin.

Known Issues

  • Workspaces become unstoppable and their statuses don’t change when the timestamps on the run’s status changes array are incorrect.

  • In the Control Center’s navigation pane, click Assets. Domino displays a "Failed to get assets" message and takes a long time to load the assets.

  • You cannot delete, rename, or move a .gitignore file in a Domino File System project.

    • If you delete the file, it remains.

    • If you move the file, the original and target file both exist.

    • If you rename the file, the source and renamed file both exist.

  • If you rerun a successful job that accesses files from a dataset in a project owned by an organization, the job fails with file not found errors. Initial runs of the jobs succeed, but reruns fail with logging similar to the following:

    2022-10-06 04:34:32 : FileNotFoundError: [Errno 2] No such file or directory: '/domino/datasets/local/reruns_fail/file.txt'

    To work around this, start the execution as a new job.

  • If you use a forward slash (/) in a Git branch name, the forward slash and any preceding text is stripped. For example, if the branch was named test/branch1, it will be renamed branch1.

    • In a workspace in a Git-based project, you can’t pull data because the branch doesn’t exist. You can push because this creates a new branch named branch1.

    • In a Domino File System project that uses imported code repositories, you can push and pull directly to and from the branch (test/branch2) if you don’t change the branch in the menu. If you change the branch, you can’t return to the selection (test/branch2) because test/ is stripped. If you push to branch2, this creates a new branch named branch2.

  • The graph views are skewed incorrectly, in Domino’s model monitoring, when normalized values are small.

  • When exporting notebooks from JupyterLab workspaces, the system shows a 403 Forbidden Error with Blocking Cross Origin message.

  • After upgrading Kubernetes, X509 certificate signed by unknown authority errors appear, and Domino fails to deploy workspaces, jobs, apps, models, and so on.

    This can occur if you upgrade to a Kubernetes version where Docker is not the default runtime. Domino depends on some Docker functionality (for managing certificate trust) that is not available in containerd without special configuration. See the cluster requirements for important notes about compatibility with Kubernetes 1.20 and above, and contact your Domino customer success team for guidance.

  • Entering a URI in the New Environment window can cause the window to go blank. This issue is resolved in Domino 5.2.0.

  • The output of the gateway/runs/getByBatchId endpoint is missing the runs and nextBatchId objects. This issue is resolved in Domino 5.2.0.

  • Sync operations for deployments backed by SMB file stores can sometimes result in unintended file deletions. This issue is fixed in Domino 5.2.0.

  • You cannot delete a .gitignore file in a Domino File System (DFS)-based project. This issue is fixed in Domino 5.2.0.

  • When you set an environment as the default on the Environments page and later restart the service, Domino reverts to the default environment set on the Central Configuration page.

  • Jobs can fail or become stuck in a Pending state when using files from an S3 data source. This issue is fixed in Domino 5.2.0.

  • Apps may fail with unusual error messages. This issue is resolved in Domino 5.2.0.

  • The Domino UI may hang with a "Waiting for available socket" error. This issue is resolved in Domino 5.2.0.

  • When an API key explicitly passed as an api_key argument and the DOMINO_TOKEN_FILE environment variable is set, Domino uses the token authentication. This issue is resolved in Domino 5.2.0.

  • When Flask apps generate links, this can result in unwanted 302 redirect behavior. This issue is fixed in Domino 5.2.0.

  • When a Model API fails while retrieving files from git using SSH keys, the private key and the key password, if any, may be exposed in the Domino log. This issue is resolved in Domino 5.2.0.

  • The Import Project drop-down does not display projects owned by organizations of which you are a member. This issue is resolved in Domino 5.2.0.

  • MATLAB environment builds may fail, or MATLAB workspaces may be unresponsive.

  • You might experience slow loading times when you open the Environments, Workspaces, Model APIs, or Projects pages.

  • When starting a workspace, the application sometimes gets stuck in the Pulling status, even though the logs indicate that the workspace started successfully. If this occurs, stop and start the workspace again. This issue is fixed in Domino 5.2.0.

  • The Kolmogorov–Smirnov Test, Wasserstein distance, and Energy Distance tests can lead to unpredictable results. Do not use these tests.

  • When you add environment variables to a project, they are visible in the execution_specifications collection in MongoDB.

  • After upgrading to 4.6.4 or higher, launching a job from a Git-based project that uses a shared SSH key or personal access token results in a Cannot access Git repository error.

  • In a Git-based project, requirements.txt must be in your project’s Artifacts folder in order to be picked up by an execution, instead of at the root level in your project files.

    Tip
    		 To avoid falling out of sync with the Git repository, always modify requirements.txt in the repository first and then copy it to your project’s Artifacts folder.

  • After you migrate from 4.x, scheduled jobs will fail until the users that own them log out and log in again. This issue is resolved in Domino 5.3.0.

    Tip
    		 You can also resolve this issue by removing all existing tokens in MongoDB, like this: db.user_session_values.remove({"key": "jwt"})

  • Domino can become inaccessible after a fixed uptime period when running on Kubernetes 1.21 or higher. See this knowledge base article for the workaround. This issue is fixed in Domino 5.3.1.

  • Upgrading to this release from Domino 3.x or 4.x can cause Git credentials to be omitted from the dropdown when creating a new Git-based project. A workaround is available from Domino’s Customer Success team.

  • Model Monitoring data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry will go through. However, you won’t be able to see metrics for that entry because the name points to an invalid bucket.

  • If the Model Access Token Vault migration runs more than once, it wipes tokens from Mongo and Vault, and users need to regenerate their Model Access Tokens in the UI. This is fixed in Domino 5.4.0.

  • Configuring prediction when setting up prediction capture for Model APIs causes predictions to not be captured. This issue is resolved in 5.4.0.

  • When restarting a Workspace through the Update Settings modal, External Data Volumes are not mounted in the new Workspace. Follow the steps to mount External Data Volumes. This issue is fixed in Domino 5.9.0.

  • Viewing dataset files in an Azure-based Domino cluster may lock files, preventing them from being deleted or modified. Restarting Nucleus frontend pods will release the lock. This issue is fixed in Domino 5.11.1.