Domino 5.2.0 (June 2022)

Kubernetes - see the Kubernetes compatibility chart

Ray - 1.12.0

Spark - 3.2.1

Dask - 2022.04.2

Dask ML - 2021.11.30

MPI - 4.1.3

Model publishing UX refactoring

When installing Python/R packages in your Domino environments, you no longer need to run the installation commands as root. Only OS-level commands (such as apt-get install …​) must be run as root.

The Rerun with latest job launch shortcut is now called Re-run with current version and uses the current active revision of the environment for both the main execution and the cluster environment. This is a change from the previous behavior where the project’s current default environment and revision were used for the main execution.

Model API image builds using Forge (ImageBuilder version 2) now use Repocloner (a built-in binary, not a service) instead of Replicator for cloning project files. You can revert to the previous behavior (using Replicator) by setting the ShortLived.RepoclonerImageBuilds feature flag to false.

You can use the new com.cerebro.domino.repocloner.javaOpts Central Configuration key to adjust the maximum size of memory heap. You can also add Java options to the value. The Repocloner replaces the Replicator for model builds. See Central Configuration for details.

DatasetRw

Environments

Jobs

Organizations

ProjectSharedDatasets

Projects

Users

When exporting notebooks from JupyterLab workspaces, the system shows a 403 Forbidden Error with Blocking Cross Origin message.

Now the Import Project drop-down includes projects owned by organizations of which you are a member. This fixes a known issue in Domino 4.4, 4.5, and 4.6.

Fixed an issue that prevented users from entering a URI in the New Environment window.

Domino now honors any API key explicitly passed as an api_key argument, even when the DOMINO_TOKEN_FILE environment variable is set. Likewise, it honors any token passed as a token argument, even when the DOMINO_USER_API_KEY environment variable is set.

Fixed an issue that sometimes caused the Domino application to hang with a "Waiting for available socket" error.

Fixed an issue that sometimes caused apps to fail with unusual error messages.

Fixed an issue that sometimes caused jobs to fail or become stuck in a Pending state when using files from an S3 data source.

When you set the default environment on the Environments page and then restart the service, Domino no longer reverts to the default environment previously set on the Central Configuration page.

Fixed an issue that prevented deleting a .gitignore file in a Domino File System (DFS)-based project.

Fixed a bug that allowed Javascript to be executed when viewing a raw HTML file while com.cerebro.domino.frontend.allowHtmlFilesToRender is false.

Flask apps can now generate links without unwanted 302 redirect behavior.

Database migrations that run during upgrades now stop when a failure occurs, instead of continuing with the remaining migrations. The last exception is stored in the database for troubleshooting purposes.

Fixed an issue that prevented workspaces from launching when the name of the organization or user that owned the project started with data.

The gateway/runs/getByBatchId endpoint now produces the correct output, including the runs and nextBatchId objects.

Fixed an issue that caused file deletions during sync operations for deployments backed by SMB file stores.

Fixed an issue that caused MATLAB environment builds to fail and MATLAB workspaces to become unresponsive.

Fixed an issue that caused the application to get stuck displaying the Pulling status after starting a workspace.

URLs in the Job Details tab will no longer have single quotes, which prevented users from viewing the job file.

Workspaces become unstoppable and their statuses don’t change when the timestamps on the run’s status changes array are incorrect.

After deleting branches that were created in GitHub but not pulled (that is, they are not local on Domino), they are still shown in the Code menu in the Data tab of the workspace.

If you are comparing jobs that have large numbers of output files, you might encounter a 502 Bad Gateway error.

In the Control Center’s navigation pane, click Assets. Domino displays a "Failed to get assets" message and takes a long time to load the assets.

You cannot delete, rename, or move a .gitignore file in a Domino File System project.

If you rerun a successful job that accesses files from a dataset in a project owned by an organization, the job fails with file not found errors. Initial runs of the jobs succeed, but reruns fail with logging similar to the following:

2022-10-06 04:34:32 : FileNotFoundError: [Errno 2] No such file or directory: '/domino/datasets/local/reruns_fail/file.txt'

If you use a forward slash (/) in a Git branch name, the forward slash and any preceding text is stripped. For example, if the branch was named test/branch1, it will be renamed branch1.

The graph views are skewed incorrectly, in Domino’s model monitoring, when normalized values are small.

ImageBuilder V3 does not correctly install or function when installing Domino 5.2.0 in these environments:

If you have the configuration key refreshTokenInRun set to true and are migrating from versions 4.3.x through 4.6.x to 5.x, Scheduled Jobs will no longer work. JWT tokens generated prior to Domino 5.x are not compatible with the offline tokens used in Domino 5.x. To remedy this, individual users must logout of Domino, then log back in. Administrators can use the following command to remove all existing tokens from MongoDB:

db.user_session_values.remove({"key": "jwt"})

Due to a bug in Rancher 2.6.0 and above that makes it unable to properly create and manage TLS certificates, Domino is unable to deploy and validate Kubernetes >= 1.21 for Rancher deployments.

You might experience slow loading times when opening the Environments, Workspaces, Model APIs, or Projects pages.

You might experience slow response times with the Projects API, Projects v4 API, and the /v4/gateway/projects/findProjectByOwnerAndName endpoint.

The Kolmogorov–Smirnov Test, Wasserstein distance, and Energy Distance tests can lead to unpredictable results. Do not use these tests.

The lists of projects and scheduled runs might load slowly when you have many projects and models.

When a node runs low on available memory, Kubernetes might evict job and workspace pods residing on that node, and those pods might fail to start up on a different node. To minimize pod eviction, set the hardware tier memory limit to be equal to the memory request.

Setting com.cerebro.domino.workloadNotifications.isRequired to true does not prevent users from turning off notifications, and changing the value of com.cerebro.domino.workloadNotifications.maximumPeriodInSeconds has no effect.

When you add environment variables to a project, they are visible in the execution_specifications collection in MongoDB.

After you upgrade to 4.6.4 or higher, if you launch a job from a Git-based project that uses a shared SSH key or personal access token, it results in a Cannot access Git repository error.

In a Git-based project, requirements.txt must be in your project’s Artifacts folder in order to be picked up by an execution, instead of at the root level in your project files.

When submitting a large number of jobs, Domino processes the first batch of jobs as usual but if the total number of jobs is larger than the execution quota for the user or the hardware tier, then the queued jobs are not promptly processed when capacity is available under the quota. This issue is resolved in Domino 5.2.2.

After you migrate from 4.x, scheduled jobs will fail until the users that own them log out and log in again. This issue is resolved in Domino 5.3.0.

Domino can become inaccessible after a fixed uptime period when running on Kubernetes 1.21 or higher. See this knowledge base article for the workaround. This issue is fixed in Domino 5.3.1.

Upgrading to this release from Domino 3.x or 4.x can cause Git credentials to be omitted from the dropdown when creating a new Git-based project. A workaround is available from Domino’s Customer Success team.

When you use drag-and-drop to upload a large number of files at once, the upload might fail. The application displays many empty dialogs. Use the Domino CLI to upload large numbers of files instead. Domino 5.4.0 eliminates the empty dialogs and shows helpful information.

Model Monitoring data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry will go through. However, you won’t be able to see metrics for that entry because the name points to an invalid bucket.

A limit overcommit error (OOMKilled) is occurring frequently on the prometheus-adapter pod in Domino’s Integrated Model Monitoring. The current workaround is to increase the limits. This issue is resolved in Domino 5.5.0.

If the Model Access Token Vault migration runs more than once, it wipes tokens from Mongo and Vault, and users need to regenerate their Model Access Tokens in the UI. This is fixed in Domino 5.4.0.

"Other" Git service provider URLs not ending in ".git" fail to work when you attempt to create a project from a Git repo or add a Git repo to a project. This issue is fixed in Domino 5.5.0.

Configuring prediction when setting up prediction capture for Model APIs causes predictions to not be captured. This issue is resolved in 5.4.0.

When restarting a Workspace through the Update Settings modal, External Data Volumes are not mounted in the new Workspace. Follow the steps to mount External Data Volumes. This issue is fixed in Domino 5.9.0.

Viewing dataset files in an Azure-based Domino cluster may lock files, preventing them from being deleted or modified. Restarting Nucleus frontend pods will release the lock. This issue is fixed in Domino 5.11.1.