See also the fleetcommand-agent Release Notes.
The following versions have been validated with Domino 5.11.0. Other versions might be compatible but are not guaranteed.
-
Kubernetes - see the Kubernetes compatibility chart
Qdrant Vector DB Data Source (GA)
The new Qdrant vector database Data Source connector enables easy, secure access to vectorized content stored in Qdrant. Domino Data Sources provide enhanced security and governance capabilities while supporting cutting-edge Generative AI use cases.
FinOps Billing Tags and Nexus support (Private preview)
-
Cost Centers and Organizations can now be used within Domino as Billing Tags for Showbacks and Chargebacks.
-
Domino’s FinOps feature extends support to Nexus data planes for comprehensive cost analysis.
Domino Flows (Public preview)
Domino Flows enables you to efficiently orchestrate and monitor complex, multi-step computations while ensuring full lineage and reproducibility. See the Flows documentation for more details.
GPU Resource Monitoring (GA)
You can now monitor GPU utilization and memory usage for Workspaces, Jobs, and any attached compute clusters running on GPU-enabled Hardware Tiers. This feature provides detailed insights into both current and historical resource usage, enhancing your ability to manage and optimize GPU resources effectively.
-
The Domino AI Gateway is now accessible through the user interface (UI) under the Endpoints tab, allowing for easy access and management of Gateway large language models.
-
The Domino UI now fully supports locale-based date formats, allowing users to view dates in the format specified by their browser settings.
-
The accessibility of the Domino UI has been enhanced, with most pages achieving Lighthouse scores of 90 or above for accessibility.
-
The headings of the Jobs table have been revamped, transitioning from icon-based representations to textual descriptions, specifically TOTAL DURATION, STATUS, and COMMENTS. Additionally, the STATUS display has undergone an upgrade, shifting from a color-based system to a more comprehensive display featuring colored icons accompanied by descriptive text.
-
The Jobs table has been optimized to ensure accurate and efficient updates, even in scenarios where there are substantial changes.
-
The Markdown display has been enhanced to support GitHub Flavored Markdown, mathematical formulas, and HTML syntax. The HTML is rendered when the configuration flag
com.cerebro.domino.frontend.allowHtmlFilesToRender
is set totrue
.
-
The Pinecone vector database Data Source connector can now be accessed through Python pinecone-client >= 3.0.0.
-
Workspace Quality of Life Improvements:
You can now use Sync All to quickly sync your changes to DFS or your imported repository. You will be able to enter a commit message if required and resolve any detected merge conflicts seamlessly.
The workspace’s side navigation panel now uses a toggle button, located at the bottom of the panel, to facilitate the opening and closing of the panel.
Tooltips in the File Changes tab are now more descriptive.
-
You can now log in to Diagnostics Toolkit using your Domino credentials. A link to navigate to Diagnostics Toolkit has been added to the Advanced menu in the Admin Area.
-
Batch file uploading is now used when uploading a large number of files in the UI. You can configure the batch size on the Central Configuration page using
com.cerebro.domino.frontend.uploadFileBatchSize
. The default batch size is 512 files.
-
Users can now create review requests for forked project updates that include
.ipynb
files without the review page becoming unresponsive.
-
The Shiny App included in the quick-start project won’t be functional out of the box with the Domino Core Environment (DCE) because the DCE image does not include the Shiny package. To work around this issue, add the following instruction to the DCE Dockerfile:
RUN R --no-save -e "install.packages(c('shiny'))"
.
-
A small number of Domino executions will fail due to a transient issue in the underlying Kubernetes API version 1.28.3. If you encounter an execution failure with the error message,
MountVolume.SetUp failed for volume 'execution-secrets-vol': failed to sync secret cache
, retry the execution to attempt to resolve the error.
-
S3 buckets must have CORS enabled to use the View Latest Raw File button in the code file browser if the file is > 5 MB (
com.cerebro.domino.frontend.defaultMaxFileSizeToRenderInBytes
). As a workaround, use the Download button to download larger files and view them on your computer.
-
In Azure Blob Store deployments, Projects with many files may fail to sync through the Domino CLI. To work around this issue, do not disable file locking when prompted by Domino.
-
You cannot view the latest raw file if you click View Latest Raw File. In the navigation pane, go to Files and click a file to view its details.
-
When uploading a large file to the Azure blob store by syncing a Workspace, you may encounter a Java Out of Memory error from Azure if the file/blob already exists. To work around this issue, use the Domino CLI to upload the file to the Project.
-
Model Monitoring data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry will go through. However, you won’t be able to see metrics for that entry because the name points to an invalid bucket.
-
Domino instances that make use of Azure Blob Storage may experience stalled Jobs within Projects with many large files.
-
If you attach a Git repository to a DFS Project that points to a tagged release, the tag won’t be honored when building a model API in that Project. The build log will show an error similar to the following, and the model will be built using the default branch of your Git repository instead of the tagged branch:
Jul 05 2023 14:36:27 -0500 #10 6.481 WARN [d.r.d.GitRepoUpdater] could not parse ref: v1.3.0 checking out default branch correlationId="iA2qWrYSLQ" thread="main"
To work around this issue, use the branch name when building model APIs instead of the release tag.
-
If an admin resets a user’s password, it invalidates all the user’s authentication tokens, including tokens used for long-running tasks like Jobs, Workspaces, or Apps. The user must create a new password, log back into Domino, and restart all executions. This also applies to CLI authentication; the user must re-login to their Domino CLI.
-
In Domino 5.6, the cost analyzer pod (inactive unless Kubecost is enabled) defaults to a different
storageClass
compared to Domino 5.7. As a result, the pod won’t run after upgrading to 5.7, breaking Kubecost functionality. However, data will continue to persist in Prometheus (or custom storage if using Kubecost Enterprise).To prevent this issue while still in Domino 5.6, override the default storageClass
gp2
with the one expected in 5.7,dominodisk
, during Kubecost installation by settingrelease_overrides.cost-analyzer.chart_values.persistentVolume.storageClass
todominodisk
in the agent YAML before installing Kubecost.If you’ve already installed Kubecost on Domino 5.6, avoid the upgrade error by setting
release_overrides.cost-analyzer.chart_values.persistentVolume.storageClass
togp2
in the agent YAML configuration file before upgrading to 5.7.
-
The Rename dataset’s file button is not available when the user navigates to the Dataset from the global Dataset page.
To work around this issue, navigate to the Dataset from the Project’s page.
-
The sample script for making asynchronous Model API requests contains an extra
/
at the end of theDOMINO_URL
variable. As a result, running the script will show an error similar to the following.{'requestId': 'key not found: HandlerDef', 'errors': ['java.util.NoSuchElementException: key not found: HandlerDef']}
To work around this issue, remove the trailing
/
at the end of theDOMINO_URL
variable.
-
The Jobs REST API uses
GitRefV1
to reference Git objects (commits, branches, and tags). Not all examples in the API spec worked, so they’ve been updated to reflect the actual valid values. This change doesn’t affect API functionality; it’s just a fix to the documentation.
-
Links to Stack Trace and CPU Flame Graph in the Ray Cluster UI’s Cluster tab are broken due to an issue in Ray 2.4 not supporting links when hosted behind a reverse proxy. This problem is specific to the Cluster tab; links correctly function in other tabs. The issue is fixed in Ray 2.7 and will be updated in future Domino Ray image releases.
-
The section, Account Settings > Login Profile, has been temporarily disabled for all users, resulting in users not being able to edit their username, name, email, etc. This section will be rebuilt in a future Domino release.
-
Cost-analyzer provisions a new default blob storage on S3 that will be used as default storage for AWS users who use Domino-automated infrastructure install and upgrades. This storage is created with a 15-day retention period. As a result, data that was stored in Prometheus during the upgrade may not be available on S3 but still accessible on Prometheus. Additionally, data stored in S3 will only be available for 15 days. To increase the retention period, update the S3 lifecycle.
-
Cost-analyzer provides up to 15 days of data for users without an Enterprise license. As a result, no notification will be sent after 15 days if the budget threshold has been reached. To receive notifications after 15 days, please reach out to your Domino representative.
-
Unable to connect to the Starburst-powered JDBC Data Sources if the credentials contain special characters.
To work around this issue, create the credentials without using the following special characters:
=
,%
,+
, or,
.
-
Flyteadmin pods may initially be in a CrashLoopBackoff when Domino 5.11 is installed or upgraded for the first time, while it waits for a job to create the Kubernetes secret
flyte-admin-secrets
. This situation will resolve itself automatically after some time.
-
Flows is only supported in Domino Cloud, AWS, and Azure. A future update will add support for Google Cloud and on-premises installs.
-
Stopping a Domino Job that has been triggered through Flows may incorrectly mark the Flow as successful rather than aborted.
-
The use of
.
in a Flows input name will cause a 400 error when trying to re-run the task.
-
Use of
is_latest=true
in a Flow does not work with compute clusters.
-
Some Flyte features are not yet available in Flows, such as dynamic workflows, eager workflows, and map_task.
-
In Workspaces, there is a known issue where AWS credentials fail to refresh if the Workspace operates beyond the default expiration time of 1 hour. To resolve this, you must stop and restart the Workspace when credentials expire. Alternatively, you can extend the token lifetime to a more suitable duration using the
com.cerebro.domino.auth.aws.sts.defaultSessionDuration
Central Configuration setting, provided it aligns with company IT policies to mitigate this issue effectively.
-
Viewing dataset files in an Azure-based Domino cluster may lock files, preventing them from being deleted or modified. Restarting Nucleus frontend pods will release the lock. This issue is fixed in Domino 5.11.1.
-
Jobs running for longer than 7 hours in an AWS-based Domino cluster will likely fail to save the results. This issue can be prevented by setting the Central Configuration
com.cerebro.domino.useDeprecatedAwsSdkV1: true
. This issue is fixed in Domino 5.11.1.
-
Workspace auto-deletion notifications and the deletion itself may not complete successfully. The error
"Cannot apply $addToSet to non-array field"
may be observed in nucleus-workspace-volume-snapshot-cleaner or nucleus-develop pods. This issue may prevent idle workspaces from being automatically deleted. Contact Domino Support if you need help. This is fixed in Domino 6.0.0.
-
Uploading files to Datasets on Windows using the
datasets_upload_files()
function from the dominodatalab library may fail in versions earlier than 1.4.3. This issue is fixed in Domino 6.0.0.
-
Downloading a Dataset file with a space in its name using the Download Selected Items button can fail. As a workaround, you can download the file via the kebab icon next to the filename or rename the file before downloading. This issue is fixed in Domino 6.0.0.
-
Running apps within VS Code workspaces may fail to be rendered through VS Code port proxy. This issue is fixed in 6.0.1 and 6.1.0.
-
GKE users that provisioned their infrastructure with Domino’s terraform-gcp-gke module must apply the changes introduced for
5.7.0
as of terraform-gcp-gke v2.5.0 when upgrading to ensure firewall rules work properly. -
VPN support from within executions was updated to be disabled by default. Support can be enabled by setting the global config value
com.cerebro.domino.computegrid.executions.allowVpn = true
.
-
MongoDB is no longer the authoritative source of truth for User Roles. Keycloak has taken over the role. User Groups in Keycloak now correspond to Domino Global Roles, and a user’s membership status in these groups defines their Domino roles. The Central Config key
authentication.oidc.externalRolesEnabled
has been retired and no longer has any effect. Any edits made to roles in MongoDB will be overridden by the data from Keycloak.
-
EKS users are recommended to update the AWS VPC CNI settings to enable
ANNOTATE_POD_IP
to prevent execution timeout errors when an image pull takes longer than 10 minutes. To bypass the validation check during an upgrade, pass--warn-only
as a command line option to the installer.
-
EKS users who provisioned their infrastructure with Domino’s terraform-aws-eks module must validate whether they want the new costs blob storage to be provisioned by default.
-
For customers with an XL-sized deployment that are upgrading to 5.9.0, RabbitMQ and New Relic resource requirements have increased.
-
Upgraded PostgreSQL to 14.10.
-
Domino CLI clients version 1.x (released in 2017 or earlier) are no longer supported. It is recommended to upgrade to Domino CLI version 6.0.