About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Domino 5.3.0 (September 2022)

See also the fleetcommand-agent Release Notes.

Validated Frameworks

The following versions have been validated with Domino 5.3.0. Other versions might be compatible, but are not guaranteed.

Upgrade notes

Important
 Before upgrading to this release, you must put Domino into maintenance mode to avoid losing work. Maintenance mode pauses all apps, model APIs, restartable workspaces, and scheduled jobs. Allow running jobs to complete or stop them manually.

New Features

Support for complex inference processing using GPUs

You can now use GPUs to achieve faster speeds and larger batches when handling complex inference processing on deep-learning models. When you deploy a model, you can configure each node that hosts the Model API with its own GPU instances. See Publish a Model and Create Model Resource Quotas.

New Starburst-powered data sources

Administrators can create Starburst-powered data sources that Domino users can access to connect to these data store types:

Note
 Domino must enable this feature. Contact your Customer Success engineer.

New Trino data source

Now you can connect to any Trino-connected data store from Domino. This gives you access to many types of distributed data sources for your data science projects, regardless of whether Domino has a dedicated connector for them. See the list of Trino connectors to learn about the supported data store types.

See Connect to Trino for how to create a Trino data source in Domino.

Audit logging

For good practice (GxP) compliance, Domino created the Project Permissions Log to help you audit who has access to sensitive data, such as blind data for a clinical trial’s statistical analysis and reporting. You can download this non-editable report that lists the users whose access to specific projects has been granted or revoked. This ensures the integrity of your project by confirming that blind data has not been compromised. See Download the Audit Log.

New roles

Domino offers these new purely administrative user roles that don’t have access to potentially-sensitive data:

  • Limited Admin: A user with this role can only perform the following actions:

    • Stop jobs in public projects

    • All administrator actions _except:

      • Edit the Central Configuration

      • Edit users

      • Create or edit global environments

      • Run MongoDB queries

  • License Reviewer: A user with this role can only generate usage reports, with no access to other functionality.

See Roles for details.

Improvements

  • Updated environments provide the latest versions of third-party tools and address security vulnerabilities:

    • Domino Standard Environment (DSE): quay.io/domino/compute-environment-images:ubuntu20-py3.9-r4.2-domino5.3-standard

    • Domino Minimum Environment (DME): quay.io/domino/compute-environment-images:ubuntu20-py3.9-domino5.3-minimal

    • Domino Spark Environment: quay.io/domino/compute-environment-images:ubuntu20-py3.9-r4.2-spark3.3.0-hadoop3.3.3-domino5.3

    • Domino Ray Environment: quay.io/domino/compute-environment-images:ubuntu20-py3.9-r4.2-ray1.13-domino5.3

    • Domino Dask Environment: quay.io/domino/compute-environment-images:ubuntu20-py3.9-r4.2-dask2022.8.1-domino5.3

  • You can now rename files or folders in a dataset in the Domino application. You can also delete files and folders. In previous versions, you had to do this in a terminal session.

  • This release removes the Kolmogorov–Smirnov, Wasserstein distance, and energy distance tests because they led to unpredictable results. Existing checks continue to use these tests; however, Domino recommends that you select different tests. See Analyze Data Drift for details about the available tests.

  • Domino now provides simplified domino.yaml configuration files for faster and smoother installs.

  • The Server Message Block protocol (SMB) driver is now disabled by default. If you want to use an external data volume that is backed by an SMB persistent volume, add the following to your installer configuration and then run the installer.

    release_overrides:
  csi-driver-smb:
    installed: true

  • ImageBuilder V2 (Forge) is no longer supported. ImageBuilder V1 (Legacy) will no longer be supported in Domino v5.4.0. Use ImageBuilder V3 (Hephaestus).

  • Docker bridge is not supported in Kubernetes 1.24 and above.

API changes

This release introduces new public REST API endpoints, an updated python-domino library, and support for R in the Domino Data API.

New public REST API endpoints

These REST API endpoints are publicly available in this release:

See REST API Reference for details.

New methods in python-domino

The python-domino library is updated to version 1.1.1 and includes the following new methods. See The python-domino Library for complete reference information.

Methods used to manage project tags
Methods used to manage datasets

Domino Data API for R

The Domino Data API is now available for R in addition to Python. See Install the Data API for installation instructions. This version of the Domino Data API for R supports data source use cases only.

You can find the public repository for the Domino Data API for R at the DominoDataR repository.

Bug Fixes

  • If you are comparing jobs that have large numbers of output files, you can use the following Central Configuration keys to set limits if you encounter a 502 Bad Gateway error.

    • com.cerebro.domino.runResults.maximumNumberOfInputComparisons (default:1000)

    • com.cerebro.domino.runResults.maximumNumberOfInputDiffs (default: 250)

    • com.cerebro.domino.runResults.maximumNumberOfResultsComparisons (default: 1000)

    • com.cerebro.domino.runResults.maximumNumberOfResultsDiffs (default: 250) If too many files are compared, you might have to increase the request timeout. See Increase Request Timeout to Compare Jobs.

  • You can now delete, rename, or move a .gitignore file in a Domino File System project.

  • You can rerun a successful job that accesses files in a dataset in a project that is owned by an organization. The job previously failed with file not found errors.

  • You can now change the Visibility of a compute environment from Globally accessible to private.

  • Fixed 403 Forbidden Error with Blocking Cross Origin message that occurred when exporting notebooks from JupyterLab workspaces.

  • Fixed Control Center to load the Compute & Spend page without timing out.

  • Workspace jobs in Kubernetes are not started if the username is incorrect. For example, if a username has the @ symbol, such as jayne@example.com.

  • Even if an administrator configures Intercom incorrectly, the Domino application will now load properly.

  • Builds now complete when using Hephaestus with Istio disabled.

  • If you migrate from 4.6.4 to 5.3.0, refresh tokens are no longer left in the database. Scheduled jobs will continue to run.

  • Domino’s Model Monitoring (DMM) application now shows all bins from the raw data. Labels are also no longer truncated.

  • When you start an execution and check tooling-jwt and tooling-aws, they both now have the proper paths that were propagated through the secret in the readiness checks.

  • When you specify multiple registries in com.cerebro.domino.builder.remoteRegistryCredentials.additionalServers, you can create builds for each registry. This resolves a problem in Image Builder V3, where if one registry was used to create the first build, you couldn’t use other registries in subsequent builds.

  • The requirements.txt file is now auto-installed from the Code directory in Git-based projects. Previously users had to move the file to the Artifacts directory.

Known Issues

  • Workspaces become unstoppable and their statuses don’t change when the timestamps on the run’s status changes array are incorrect.

  • You cannot view the latest raw file through the UI. In the navigation pane, go to Files and click a file to view its details. If you click View Latest Raw File, a blank page opens. Workaround: Download the file using the "Download" button and view it on your computer.

  • After deleting branches that were created in GitHub but not pulled (that is, they are not local on Domino), they are still shown in the Code menu in the Data tab of the workspace.

  • In the Control Center’s navigation pane, click Assets. Domino displays a "Failed to get assets" message and takes a long time to load the assets.

  • In projects where the Environment variables and Files checkboxes are selected in the Exports settings, you might see a 500 page console error Failed to load resource when you create and save a new file.

  • If you use a forward slash (/) in a Git branch name, the forward slash and any preceding text is stripped. For example, if the branch was named test/branch1, it will be renamed branch1.

    • In a workspace in a Git-based project, you can’t pull data because the branch doesn’t exist. You can push because this creates a new branch named branch1.

    • In a Domino File System project that uses imported code repositories, you can push and pull directly to and from the branch (test/branch2) if you don’t change the branch in the menu. If you change the branch, you can’t return to the selection (test/branch2) because test/ is stripped. If you push to branch2, this creates a new branch named branch2.

  • The graph views are skewed incorrectly, in Domino’s model monitoring, when normalized values are small. If you create a new GitLab repository during creation of a Git-based project, Domino shows the following message: Cannot create project because of Git repository creation failure - Credential does not have permission to create a repository. However, the repository is created. GitLab has a bug that causes this message. See Create a Git-based Project.

  • Domino can become inaccessible after a fixed uptime period when running on Kubernetes 1.21 or higher. See this knowledge base article for the workaround. This issue is fixed in Domino 5.3.1.

  • Domino does not accept periods in Snowflake data source account names. This issue is fixed in Domino 5.3.1.

  • Upgrading to this release from Domino 3.x or 4.x can cause Git credentials to be omitted from the dropdown when creating a new Git-based project. A workaround is available from Domino’s Customer Success team.

  • When you use drag-and-drop to upload a large number of files at once, the upload might fail. The application displays many empty dialogs. Use the Domino CLI to upload large numbers of files instead. Domino 5.4.0 eliminates the empty dialogs and shows helpful information.

  • Model Monitoring data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry will go through. However, you won’t be able to see metrics for that entry because the name points to an invalid bucket.

  • A limit overcommit error (OOMKilled) is occurring frequently on the prometheus-adapter pod in Domino’s Integrated Model Monitoring. The current workaround is to increase the limits. This issue is resolved in Domino 5.5.0.

  • Runs might fail during startup with an error about a missing Kerberos environment variable. This issue is resolved in Domino 5.3.2.

  • Nucleus crashes when you go to Admin > Executions on a deployment with hundreds of executions. This issue is resolved in Domino 5.3.2.

  • When a practitioner user attempts to update a project forked from another after the source project has been updated, a 403 Access Forbidden error is shown.

  • Domino Model Monitor is not compatible with Kubernetes versions 1.21 and above and stops working after 90 days of uptime. A periodic graceful restart of Domino Model Monitor is advised as a workaround. This issue is resolved in Domino 5.5.0.

  • If the Model Access Token Vault migration runs more than once, it wipes tokens from Mongo and Vault, and users need to regenerate their Model Access Tokens in the UI. This is fixed in Domino 5.4.0.

  • "Other" Git service provider URLs not ending in ".git" fail to work when you attempt to create a project from a Git repo or add a Git repo to a project. This issue is fixed in Domino 5.5.0.

  • If the nucleus-dispatcher Kubernetes pod is restarted (during Domino upgrade, after restarting Nucleus services via the admin central configuration page, after the previous pod crashing, or for some other reason), then existing executions (including workspaces and jobs) may fail.

  • Configuring prediction when setting up prediction capture for model APIs causes predictions to not be captured. This issue is resolved in 5.4.0.

  • When restarting a Workspace through the Update Settings modal, External Data Volumes are not mounted in the new Workspace. Follow the steps to mount External Data Volumes. This issue is fixed in Domino 5.9.0.

  • Viewing dataset files in an Azure-based Domino cluster may lock files, preventing them from being deleted or modified. Restarting Nucleus frontend pods will release the lock. This issue is fixed in Domino 5.11.1.