About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Domino 5.4.0 (December 2022)

See also the fleetcommand-agent Release Notes.

Validated Frameworks

The following versions have been validated with Domino 5.4.0. Other versions might be compatible, but are not guaranteed.

New Features

New MongoDB data connector

Now you can access your MongoDB data store from Domino, using a Starburst-powered connector.

An administrator must create MongoDB data source before users can access it.

New permissions for datasets

Note
 The /api/datasetrw/v1/datasets public API endpoint was deprecated in favor of /api/datasetrw/v2/datasets. Because of implications due to the nature of the dataset permissioning change, the v1 endpoint will no longer work. See the DatasetRw REST API Guide for more.

Datasets now have a simple security model that is independent from projects. In this model, roles determine the permissions that you and your users have when you work with datasets. See Dataset Roles for details.

See the following topics that were updated for relevant information:

Support for Azure blob store

When deployed in Azure, Domino can store long-term, unstructured data in Azure blob storage buckets. You can configure the blob store in Domino’s Central Configuration.

Fleetcommand auto-registration

Domino instances can now be automatically registered in Fleetcommand. If you are automatically registered, you receive an installation token and use the Fleetcommand CLI to install Domino. In this new installation process, you don’t need to install fleetcommand-agent, Docker, kubectl, or Helm to install Domino, and you don’t need quay.io credentials to pull the Domino image. This new process supports Domino versions 4.1 and up.

The older installation process is still available if you don’t have an installation token.

Configurable file handlers

You can add or customize file handlers to modify the built-in Domino behavior for certain files and add support for new file types. Administrators can toggle this feature on or off through the new feature flag, com.cerebro.domino.executableTarget.enabled. This feature is disabled by default.

Audit trail/dataset permission changes

Audit logs now show changes to datasets:

  • When a user marks a dataset for deletion

  • When a user restores or deletes a dataset

  • When a user creates a dataset

  • When a user mounts or unmounts a shared dataset

  • When a user updates Dataset Roles and permissions

Custom metrics for model performance and drift

The new Custom Model Monitoring Metrics SDK lets you create a custom metric, define the logic to compute them, and use them alongside the out-of-the-box drift and model quality metrics that are produced by the Model Monitor. You can author this logic and evaluate the output from within a Domino project. Once defined, the code can be deployed using scheduled jobs to compute and alert on a periodic basis.

See Custom Model Monitoring Metrics for details.

Improvements

  • The Grafana dashboards include new dashboards for better observability of Kubernetes pods and clusters.

  • Domino 5.4.0 fully supports ImageBuilder V3 (Hephaestus). ImageBuilder V1 (Legacy) and V2 (Forge) have been removed.

  • Domino simplified how a user modifies a Launcher’s JSON code. Instead of two Edit buttons, the user now clicks Edit, then Switch to JSON Edit Mode. See Copy Launcher Definitions for details.

  • You can now ingest and process a model’s training, prediction, and ground truth data from Generic S3 data sources. See Connect a Data Source.

  • Pod security policies (PSPs) are no longer created in Domino installs. Instead, all pods define an explicit security context. This removes the need for duplicate modification of security contexts during pod creation by PSPs. No replacement will be added for runtime policy enforcement. Contact your Domino representative for more information.

  • The following services have been removed:

    • spark-operator

    • forge

    • replicator

    • k8s-event-pump

    If any workloads depend on these services, contact Domino for migration help.

  • Docker bridge is not supported in Kubernetes 1.24 and above.

API changes

  • The new Custom Metrics SDK enables custom metrics for monitoring model performance and drift.

Deprecations

  • The enable_pod_security_policies and create_restricted_pod_security_policy installation configuration keys are deprecated. Now, PSPs are always disabled.

Bug Fixes

  • The data plane agent will restart if an OutOfMemory exception is encountered in a thread.

  • The JVM heap size can now be customized for the data plane agent.

  • The data plane agent will detect conditions which indicate that it has been logged out from RabbitMQ and reauthenticate within 60 seconds.

  • The data plane agent will detect when its queues are deleted in RabbitMQ and restart automatically. After restart it will wait for queue re-creation by the Dispatcher.

  • The Workspace State Machine uses the run object in MongoDB to stay in sync with the execution in the case that RMQ messages are missed. It no longer uses timestamps to infer the reasonableness of a state transition.

  • After deleting branches that were created in GitHub but not pulled (that is, they are not local on Domino), they are no longer shown in the Code menu in the Data tab of the workspace. See Change Branches in the Workspace.

  • You can use a forward slash (/) in a Git branch name, whether you use a Git repository or a Domino File System project with imported code repositories. For example, you can name a branch test/branch1.

  • In projects where the Environment variables and Files checkboxes are selected in the Exports settings, you can create and save files without a 500 page console error.

  • The Assets Portfolio can now load without consistent error messages and delays. You might encounter error messages and delays in some circumstances.

  • When you use drag-and-drop to upload a large number of files at once, Domino displays a recommendation to use the CLI instead of displaying empty dialogs.

  • The graph views for model monitoring now display the correct data when normalized values are small.

  • Git credentials are now included in the menu when you create a new Git-based project after upgrading to this release from Domino 3.x or 4.x.

  • Practitioner users are able to update a project fork, applying new changes from the source project, without generating a 403 Access Forbidden error.

  • Tokens are no longer wiped from Mongo and Vault if the Model Access Token Vault migration runs more than once, so users no longer need to regenerate their Model Access Tokens in the UI after such an event.

  • Prediction probability can now be configured when setting up prediction capture for Model APIs.

Known Issues

  • You cannot view the latest raw file. In the navigation pane, go to Files and click a file to view its details. If you click View Latest Raw File, a blank page opens.

  • Running a large number of jobs concurrently (> 1000) can introduce system instability and result in some job failures. We now document how to properly size Domino resources in deployments that expect this level of load. See Sizing the Domino platform: Large.

  • When uploading a large file to the Azure blob store by syncing a workspace, you may encounter a Java Out of Memory error from Azure if the file/blob already exists. To work around this issue, use the Domino CLI to upload the file to the project.

  • Model Monitoring data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry will go through. However, you won’t be able to see metrics for that entry because the name points to an invalid bucket.

  • A limit overcommit error (OOMKilled) is occurring frequently on the prometheus-adapter pod in Domino’s Integrated Model Monitoring. The current workaround is to increase the limits. This issue is resolved in Domino 5.5.0.

  • The data plane agent does not apply backpressure to incoming RabbitMQ RPC requests. On large deployments this could cause an OutOfMemory exception shortly after start following an extended period of downtime if a substantial number of messages are queued. Restarting the data plane agent pod typically resolves this situation.

  • The agent in Domino Nexus deployments may fail due to an out-of-memory (OOM) error. This issue is resolved in Domino 5.4.1.

  • The installer upgrade validation check for the MongoDB upgrade path is incorrectly set to MongoDB 4.4 instead of 4.2. This issue is resolved in Domino 5.4.1.

  • Domino Model Monitor is not compatible with Kubernetes versions 1.21 and above and stops working after 90 days of uptime. A periodic graceful restart of Domino Model Monitor is advised as a workaround. This issue is resolved in Domino 5.5.0.

  • In Azure Blob Store deployments, projects with many files may fail to sync through the Domino CLI. To work around this issue, do not disable file locking when prompted by Domino.

  • "Other" Git service provider URLs not ending in ".git" fail to work when you attempt to create a project from a Git repo or add a Git repo to a project. This issue is fixed in Domino 5.5.0.

  • Domino instances that make use of Azure Blob Storage may experience stalled jobs within projects with many large files.

  • If the nucleus-dispatcher Kubernetes pod is restarted (during Domino upgrade, after restarting Nucleus services via the admin central configuration page, after the previous pod crashing, or for some other reason), then existing executions (including workspaces and jobs) may fail.

  • The Status, Active Version, and Owner columns do not appear in the Model API list. This issue is fixed in Domino 5.8.0.

  • When restarting a Workspace through the Update Settings modal, External Data Volumes are not mounted in the new Workspace. Follow the steps to mount External Data Volumes. This issue is fixed in Domino 5.9.0.

  • Downloading single files from Datasets will fail if the filename contains special characters, including + and &. As a workaround, remove the mentioned special characters by renaming the file. This issue is fixed in Domino 5.10.0.

  • Spaces in ADLS filenames are not allowed when getting and putting objects in Azure Data Sources with DominoDataR. As a workaround, upgrade to DominoDataR version 0.2.4. This issue is fixed in Domino 5.10.0.

  • Viewing dataset files in an Azure-based Domino cluster may lock files, preventing them from being deleted or modified. Restarting Nucleus frontend pods will release the lock. This issue is fixed in Domino 5.11.1.

Upgrade notes

Important
 If you are upgrading to this Domino release from a version earlier than Domino 5.3.0, you must put Domino into maintenance mode to avoid losing work. Maintenance mode pauses all apps, model APIs, restartable workspaces, and scheduled jobs. Allow running jobs to complete or stop them manually.