About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Domino 5.9.0 (December 2023)

See also the fleetcommand-agent Release Notes.

Validated frameworks

The following versions have been validated with Domino 5.9.0. Other versions might be compatible but are not guaranteed.

New features

CodeGen Workspace integration

Harness the power of code generation directly in your Domino Workspace. JupyterAI is pre-installed in Domino offering data scientists an AI coding assistant that can generate code, fix errors, summarize content, answer questions, and even generate entire notebooks from natural language prompts. Jupyternaut chat bot in Domino

Project goal stages

Project goals have been enhanced to drive more accountability and visibility in your Projects. You can now assign users to goals and use Project goal stages to track more granular progress towards your goals. Assign users and set goal stages

Model API Hardware Tier

You can now define Model API Hardware Tiers that are only available to Model API deployments. Tailor your hardware tiers to the unique demands of machine learning model deployment.

Nexus expanded support

Model APIs, Apps, and Launchers can now be deployed on remote data planes.

Improvements

  • Workspaces now display startup logs and startup status, keeping you informed about progress and streamlining startup troubleshooting.

  • You can configure which external data volumes get mounted to a Workspace, allowing you to reduce potential startup failures by only mounting the necessary data volumes.

  • Download multiple Dataset files and folders as ZIP or TAR archives with the new Central Config key com.cerebro.domino.dataset.batchDownloadArchiveFormat.

  • White labeled deployments can now customize the following Domino UI elements: default App URL, default support URL, and hide the Add Projects button.

  • Project goal changes now appear in the activity feed.

API changes

  • Programmatically set your AWS Billing API config with the new AWS Billing API endpoint.

Bug fixes

  • When restarting a Workspace through the Update Settings modal, External Data Volumes are correctly mounted in the new Workspace.

  • Domino instances no longer fail to start if the Require SSL setting for DominoRealm in Keycloak is set to all requests or external requests. To improve the security of our product, SSL is enabled for external requests by default.

  • Users can see raw files whose size is ⇐ 5 MB (com.cerebro.domino.frontend.defaultMaxFileSizeToRenderInBytes) when they click on the "View Latest Raw File" button in the code file browser, even if their S3 buckets don’t have CORS enabled.

Known issues

  • A small number of Domino executions will fail due to a transient issue in the underlying Kubernetes API version 1.28.3. If you encounter an execution failure with the error message, MountVolume.SetUp failed for volume 'execution-secrets-vol': failed to sync secret cache, retry the execution to attempt to resolve the error.

  • In Azure Blob Store deployments, projects with many files may fail to sync through the Domino CLI. To work around this issue, do not disable file locking when prompted by Domino.

  • You cannot view the latest raw file if you click View Latest Raw File. In the navigation pane, go to Files and click a file to view its details.

  • When uploading a large file to the Azure blob store by syncing a Workspace, you may encounter a Java Out of Memory error from Azure if the file/blob already exists. To work around this issue, use the Domino CLI to upload the file to the Project.

  • Model Monitoring data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry will go through. However, you won’t be able to see metrics for that entry because the name points to an invalid bucket.

  • Domino instances that make use of Azure Blob Storage may experience stalled Jobs within Projects with many large files.

  • If you attach a Git repository to a DFS project that points to a tagged release, the tag won’t be honored when building a model API in that Project. The build log will show an error similar to the following, and the model will be built using the default branch of your Git repository instead of the tagged branch:

    Jul 05 2023 14:36:27 -0500 #10 6.481 WARN [d.r.d.GitRepoUpdater] could not parse ref: v1.3.0 checking out default branch correlationId="iA2qWrYSLQ" thread="main"

    To work around this issue, use the branch name when building model APIs instead of the release tag.

  • If an admin resets a user’s password, it invalidates all the user’s authentication tokens, including tokens used for long-running tasks like Jobs, Workspaces, or Apps. The user must create a new password, log back into Domino, and restart all executions. This also applies to CLI authentication; the user must re-login to their Domino CLI.

  • In Domino 5.6, the cost analyzer pod (inactive unless Kubecost is enabled) defaults to a different storageClass compared to Domino 5.7. As a result, the pod won’t run after upgrading to 5.7, breaking Kubecost functionality. However, data will continue to persist in Prometheus (or custom storage if using Kubecost Enterprise).

    To prevent this issue while still in Domino 5.6, override the default storageClass gp2 with the one expected in 5.7, dominodisk, during Kubecost installation by setting release_overrides.cost-analyzer.chart_values.persistentVolume.storageClass to dominodisk in the agent YAML before installing Kubecost.

    If you’ve already installed Kubecost on Domino 5.6, avoid the upgrade error by setting release_overrides.cost-analyzer.chart_values.persistentVolume.storageClass to gp2 in the agent YAML configuration file before upgrading to 5.7.

  • The Rename dataset’s file button is not available when the user navigates to the Dataset from the global Dataset page.

    To work around this issue, navigate to the Dataset from the Project’s page.

  • The sample script for making asynchronous Model API requests contains an extra / at the end of the DOMINO_URL variable. As a result, running the script will show an error similar to the following.

    {'requestId': 'key not found: HandlerDef', 'errors': ['java.util.NoSuchElementException: key not found: HandlerDef']}

    To work around this issue, remove the trailing / at the end of the DOMINO_URL variable.

  • The Jobs REST API uses GitRefV1 to reference Git objects (commits, branches, and tags). Not all examples in the API spec worked, so they’ve been updated to reflect the actual valid values. This change doesn’t affect API functionality; it’s just a fix to the documentation.

  • Links to Stack Trace and CPU Flame Graph in the Ray Cluster UI’s Cluster tab are broken due to an issue in Ray 2.4 not supporting links when hosted behind a reverse proxy. This problem is specific to the Cluster tab; links correctly function in other tabs. The issue is fixed in Ray 2.7 and will be updated in future Domino Ray image releases.

  • The section, Account Settings > Login Profile, has been temporarily disabled for all users, resulting in users not being able to edit their username, name, email, etc. This section will be rebuilt in a future Domino release.

  • Cost-analyzer provisions a new default blob storage on S3 that will be used as default storage for AWS users who use Domino-automated infrastructure install and upgrades. This storage is created with a 15-day retention period. As a result, data that was stored in Prometheus during the upgrade may not be available on S3 but still accessible on Prometheus. Additionally, data stored in S3 will only be available for 15 days. To increase the retention period, update the S3 lifecycle.

  • Cost-analyzer provides up to 15 days of data for users without an Enterprise license. As a result, no notification will be sent after 15 days if the budget threshold has been reached. To receive notifications after 15 days, please reach out to your Domino representative.

  • Downloading single files from Datasets using the Download Selected Items button will fail if the filename contains special characters, including + and &. As a workaround, you can download these types of files via the action menu, located to the right of the filename. This issue is fixed in Domino 5.10.0.

  • Annonymous users cannot run launchers in GBP or view public GBP projects due to the git credentials migration to vault. This issue is fixed in Domino 5.9.1.

  • Spaces in ADLS filenames are not allowed when getting and putting objects in Azure Data Sources with DominoDataR. As a workaround, upgrade to DominoDataR version 0.2.4. This issue is fixed in Domino 5.10.0.

  • If a user is a collaborator on a project that contains some registered models in a model registry, and some of these models have a deployed model API, but the user doesn’t have permission to access them, then the user won’t be able to view the project’s model registry page. This issue is fixed in Domino 5.11.0.

  • Viewing dataset files in an Azure-based Domino cluster may lock files, preventing them from being deleted or modified. Restarting Nucleus frontend pods will release the lock. This issue is fixed in Domino 5.11.1.

Upgrade notes

  • There is a known issue when upgrading to 5.9.x, or doing a fresh install, with Keycloak email notifications enabled through domino.yml that will cause Keycloak’s installation to fail. The solution is to set email_notifications.enabled to false in the domino.yml during installation, then manually add the environment variables to the stateful set once the deployment is complete.

  - name: KEYCLOAK_SMTP_HOST
    value: {{ .Values.email_notifications.server }}
  - name: KEYCLOAK_SMTP_PORT
    value: {{ .Values.email_notifications.port }}
  - name: KEYCLOAK_SMTP_FROM
    value: {{ .Values.email_notifications.from_address }}
  - name: KEYCLOAK_SMTP_FROM_DISP_NAME
    value: {{ .Values.email_notifications.from_address }}
  - name: KEYCLOAK_SMTP_SSL
    value: {{ .Values.email_notifications.enable_ssl | quote }}
  - value: KEYCLOAK_SMTP_FORGOT_PASSWORD_ENABLED
    value: "true"
  - name: KEYCLOAK_SMTP_USER_NAME
    value: {{ .Values.email_notifications.authentication.username }}
  - name: KEYCLOAK_SMTP_PASSWORD
    value: {{ .Values.email_notifications.authentication.password }}

  • GKE users that provisioned their infrastructure with Domino’s terraform-gcp-gke module must apply the changes introduced for 5.7.0 as of terraform-gcp-gke v2.5.0 when upgrading to ensure firewall rules work properly.

  • VPN support from within executions was updated to be disabled by default. Support can be enabled by setting the global config value com.cerebro.domino.computegrid.executions.allowVpn = true.

  • MongoDB is no longer the authoritative source of truth for User Roles. Keycloak has taken over the role. User Groups in Keycloak now correspond to Domino Global Roles, and a user’s membership status in these groups defines their Domino roles. The Central Config key authentication.oidc.externalRolesEnabled has been retired and no longer has any effect. Any edits made to roles in MongoDB will be overridden by the data from Keycloak.

  • EKS users are recommended to update the AWS VPC CNI settings to enable ANNOTATE_POD_IP to prevent execution timeout errors when an image pull takes longer than 10 minutes. To bypass the validation check during an upgrade, pass --warn-only as a command line option to the installer.

  • EKS users who provisioned their infrastructure with Domino’s terraform-aws-eks module must validate whether they want the new costs blob storage to be provisioned by default.

  • For customers with an XL-sized deployment that are upgrading to 5.9.0, RabbitMQ and New Relic resource requirements have increased.

  • Domino CLI clients version 1.x (released in 2017 or earlier) are no longer supported. It is recommended to upgrade to Domino CLI version 6.0.