Install configuration reference

KeyDescriptionRequiredValues

schema

YAML schema version.

✓

1.14

name

Unique deployment name. This must contain the name of the deployment owner.

✓

[a-zA-Z0-9_-]+

Note
 On data plane installs, this will be the subdomain of the data plane, i.e. the data-plane of data-plane.domino.example.com.

version

Domino version to install.

✓

Supported versions: 6.1.0, 6.1.1, 6.1.2

hostname

Hostname Domino application will be accessed at.

✓

Valid fully qualified domain name (FQDN)

Note
 On data plane installs, this is the hostname of the control plane.

ssl_enabled

Should Domino only be accessible with HTTPS.

✓

true, false

ssl_redirect

Should Domino only be accessible with HTTPS.

✓

true, false

request_resources

Create Kubernetes resource requests and limits for services.

✓

true, false

enable_network_policies

Use network policies for fine-grained service access.

✓

true, false

Note
 Requires a compatible CNI plugin, for example, Calico.

external_docker_registry

A URL to ECR (Amazon Elastic Container Registry), ACR (Azure Container Registry), GCR (Google Container Registry) or GAR (Google Artifact Registry). The node must have permissions to the registry.

Note
 Either external_docker_registry or internal_docker_registry must be configured.

custom_certificates

Custom Certificates

✓

minimal_resources

Use the minimum amount of nodes and resources to run Domino. This may not be a stable configuration and should be used with caution.

false

cluster_domain

The Kubernetes cluster domain name.

cluster.local

aws_account_id

AWS account ID in which Domino is deployed.

operator_role_arn

IAM role arn.

operator_service_account_name

Kubernetes service account name.

Control and data planes

This section configures whether Domino is a "control plane" or "data plane" install.

KeyDescriptionRequiredValues

control_plane

If true, install the control plane. This currently always includes a local data plane.

true, false

data_plane.type

Which type of data plane to install. When performing a data plane-only install, this should be remote.

local, remote

Note

  • To perform a standard "control plane" install, you can omit both of these sections. The defaults are a "control plane" and a local data plane type.

  • To perform a "data plane" install, you should configure the following:

control_plane: false
data_plane:
  type: remote

Istio

This section configures how and if an Istio Service Mesh is deployed by or integrated to Domino. A Domino-deployed Istio is for Domino use only. These configurations must only be installed and/or enabled if intra-cluster encryption in transit is required.

KeyDescriptionRequiredValues

istio.enabled

Enable Istio in deployment (that is, sidecar injection).

✓

true, false

istio.install

Install Istio service with Domino.

✓

true, false

istio.cni

Configures whether Istio installation is done with a CNI.

If true, the installation is done with a CNI and requires fewer permissions; this is our preferred and recommended setting. If false, the installation will add required capabilities to every pod security policy: NET_ADMIN, and NET_RAW.

✓

true, false

Ingress controller

This section configures the NGINX ingress controller deployed by the fleetcommand-agent.

KeyDescriptionRequiredValues

ingress_controller.install

Install the NGINX ingress controller.

✓

true, false

ingress_controller.gcp_static_ip_name

On Google Cloud Platform (GCP), use a static IP address to expose nginx.

The name of the IP address

ingress_controller.class_name

Name of the ingress class for Domino.

✓

nginx

ingress_controller.istio_annotations

Include annotations related to Istio.

✓

true,false

ingress_controller.allowed_cidrs

List of allowed ingress CIDR blocks.

Namespaces

Use Namespaces to virtually segment Kubernetes executions. Domino creates namespaces according to the specifications in this section. The installer requires that these namespaces do not exist at installation.

KeyDescriptionRequiredValues

namespaces.platform.name

Namespace to place Domino services.

✓

Kubernetes Names

namespaces.compute.name

Namespace for user executions.

✓

Kubernetes Names

Note
 Might be the same as the platform namespace.

namespaces.system.name

Namespace for deployment metadata.

✓

Kubernetes Names

namespaces.istio.name

Namespace for Istio.

✓

Kubernetes Names

namespaces.*.annotations

Optional annotations to apply to each namespace.

Kubernetes Annotation

namespaces.*.labels

Optional labels to apply to each namespace.

Kubernetes Labels and Selectors

Storage classes

Storage Classes are a way to abstract the dynamic provisioning of volumes in Kubernetes.

Domino requires the following storage classes:

  1. block storage for Domino services and user executions that need fast I/O.

  2. shared storage that can be shared between multiple executions.

Domino supports pre-created storage classes, although the installer can create a shared storage class backed by NFS or a cloud NFS analog as long as the cluster can access the NFS system for read and write, and the installer can create several types of block storage classes backed cloud block storage systems like Amazon EBS.

Block

KeyDescriptionRequiredValues

storage_classes.block.create

Whether to create the block storage class.

✓

true, false

storage_classes.block.name

The block storage class name.

✓

Kubernetes Name

Note
 Always required due to platform limitations. Cannot be "" which indicates the default storage class.

storage_classes.block.type

Type of the block storage class to use.

✓

For example,

  • ebs

  • hostpath

  • gce

  • azure-disk

  • ceph-rdb

storage_classes.block.base_path

Base path to use on nodes with hostpath volumes.

storage_classes.block.default

Whether to set this storage class as the default.

✓

true, false

Shared

KeyDescriptionRequiredValues

storage_classes.shared.create

Whether to create the shared storage class.

✓

true, false

storage_classes.shared.name

The shared storage class name.

✓

Kubernetes Name

storage_classes.shared.type

Type of the shared storage class to use.

✓

For example,

  • efs

  • nfs

  • azure-file Azure File requires outbound port 445 to be open from your Azure cluster.

  • cephfs

storage_classes.shared.efs.region

EFS store AWS region.

For example, us-west-2

storage_classes.shared.efs.filesystem_id

EFS filesystem ID.

For example, fs-7a535bd1

storage_classes.shared.efs.access_point_id

Access Point ID.

For example, fsap-01234567

storage_classes.shared.nfs.server

NFS server IP or hostname.

storage_classes.shared.nfs.mount_path

Base path to use on the server when you create shared storage volumes.

storage_classes.shared.nfs.mount_options

YAML List of additional NFS mount options.

For example, - mfsymlinks

storage_classes.shared.azure_file.storage_account

Azure storage account to create file stores.

Blob storage

Domino can store long-term, unstructured data in blob storage buckets. Only the shared storage class described previously (NFS and S3) are supported for logs.

projects also support Azure storage, and backups also supports both Azure and Google Cloud Storage (GCS) storage.

S3

KeyDescriptionRequiredValues

blob_storage.*.s3.region

AWS region of the S3 bucket store.

For example, us-west-2

blob_storage.*.s3.bucket

S3 bucket name.

For example, domino-bucket-1

blob_storage.*.s3.sse_kms_key_id

KMS Key ARN.

For example, arn:aws:kms:us-west-2:890728157128:key/mrk-eb66487f9d674b46b3ca3c8b934ddd4b

Azure

KeyDescriptionRequiredValues

blob_storage.*.azure.account_name

Azure storage account name.

For example, mystorage

blob_storage.*.azure.account_key

Access key for the storage account.

For example, domino-bucket-1

blob_storage.*.azure.container

Name of the container in the storage account.

For example, backups

GCS (Google Cloud Storage)

KeyDescriptionRequiredValues

blob_storage.backups.gcs.bucket

The bucket name.

For example, domino-bucket-1

blob_storage.backups.gcs.service_account_name

The service account name with write access to the bucket.

For example, bucket-sa

blob_storage.backups.gcs.project_name

The service account’s project name.

For example, myproject

Autoscaler

For Kubernetes clusters without native cluster scaling in response to new user executions, Domino supports the use of the cluster autoscaler.

KeyDescriptionRequiredValues

autoscaler.cloud_provider

Cloud provider Domino is deployed with.

For example, aws

autoscaler.aws.region

AWS region Domino is deployed into.

For example, us-west-2

autoscaler.azure.resource_group

Azure resource group Domino is deployed into.

Azure resource group

autoscaler.azure.subscription_id

Azure subscription ID Domino is deployed with.

Azure subscription ID

AWS Auto Discovery

The cluster autoscaler supports Auto Discovery on AWS. Without any explicit configuration of specific autoscaling groups, it detects all ASGs that have the appropriate tags and refreshes them if their settings are updated directly. All ASGs must be listed with accurate min/max settings (or not listed at all) is not required as referenced below in the Groups section. ASG settings can be updated directly in AWS. The cluster-autoscaler configuration doesn’t need to be updated, and you don’t need to rerun the installer.

KeyDescriptionRequiredValues

autoscaler.auto_discovery.cluster_name

K8s Cluster Name.

Must exactly match the name in AWS

autoscaler.auto_discovery.tags

Optional: If filled in, cluster_name is ignored.

For example, - my.tag or []

autoscaler.groups

Must be set to [] if using auto_discovery

By default, if no autoscaler.groups and autoscaler.auto_discovery.tags are specified, the cluster_name will be used to look for the following AWS tags:

  • k8s.io/cluster-autoscaler/enabled

  • k8s.io/cluster-autoscaler/{{ cluster_name }}

The tags setting can be used to explicitly specify which resource tags the autoscaler service must look for.

To disable auto-discovery and use specific groups, ensure that auto_discovery.cluster_name is an empty value.

Groups

Autoscaling groups are not dynamically discovered. Each autoscaling group must be individually specified, along with the minimum and maximum scaling size.

KeyDescriptionRequiredValues

autoscaler.groups.*.name

Autoscaling group name.

Must exactly match the name in the cloud provider

autoscaler.groups.*.min_size

Minimum scaling size.

For example, 0

autoscaler.groups.*.max_size

Maximum scaling size.

For example, 10

External DNS

Domino can automatically configure your cloud DNS provider. See external-dns for more information.

KeyDescriptionRequiredValues

external_dns.provider

Cloud DNS provider.

For example, aws

external_dns.domain_filters

Only allow access to domains that match this filter.

For example, my-domain.example.com

external_dns.zone_id_filters

Only allow updates to specific hosted zones.

txt_owner_id

The owner ID in the TXT record.

Email notifications

Domino supports SMTP to send email notifications in response to user actions and run results.

KeyDescriptionRequiredValues

email_notifications.enabled

Whether Domino must send email notifications.

✓

true, false

email_notifications.server

SMTP server hostname or IP.

email_notifications.port

SMTP server port.

email_notifications.enable_ssl

Whether the SMTP server uses SSL encryption.

✓

true, false

email_notifications.from_address

Email address to send emails from Domino with.

For example, domino @example.com

email_notifications.authentication.username

If you use SMTP authentication, the username.

email_notifications.authentication.password

If you use SMTP authentication, the password.

Monitoring

Domino supports in-cluster monitoring with Prometheus as well as more detailed, external monitoring through New Relic APM and Infrastructure.

KeyDescriptionRequiredValues

monitoring.prometheus_metrics

Install Prometheus monitoring.

✓

true, false

monitoring.node_exporter

Enable Node Exporter to generate and expose hardware and OS metrics.

true, false

monitoring.kube_state_metrics

Enable Kube-State-Metrics to generate and expose cluster-level metrics.

true, false

monitoring.newrelic.apm

Enable New Relic APM.

✓

true, false

monitoring.newrelic.infrastructure

Enable New Relic Infrastructure.

✓

true, false

monitoring.newrelic.license_key

New Relic account license key.

monitoring.newrelic.managed_controlplane

Configure New Relic for Kubernetes clusters with managed control planes, i.e. EKS, AKS, and GKE.

monitoring.gpu_metrics

Enable GPU metrics.

Grafana Alerts

Domino supports in-cluster alerting with Grafana. To enable in-cluster alerting, a Grafana contact point needs to be defined to receive alert notifications.

KeyDescriptionRequired

grafana_alerts.slack.name

The display name of the Slack contact point in the Grafana UI.

grafana_alerts.slack.channel

The name of the Slack channel to send alerts to.

grafana_alerts.slack.token

The Slack app token used to authenticate to your Slack workspace.

Helm

Configuration for the Helm repository that stores Domino’s charts.

KeyDescriptionRequiredValues

helm.version

Which version of Helm to use.

✓

2 or 3

helm.host

Hostname of the chart repository.

✓

For Helm 2 this must be quay.io or the address of your private app server. For Helm 3 it must be gcr.io.

helm.namespace

Namespace to find charts in the repository.

Helm repo namespace. When you use official Domino repositories this must be domino. For Helm 3 with gcr.io or mirrors.domino.tech, use domino-eng-service-artifacts.

helm.username

Username for chart repository if authentication is required. When you use Helm 3 with charts hosted in GCR this must be _json_key.

Username

helm.password

Password for chart repository if authentication is required.

For Helm 3 this is the base64 encoded JSON key that was provided by Domino.

Image registries

List of Docker registries for Domino components.

KeyDescriptionRequiredValues

helm.image_registries.*.server

Docker registry host.

✓

For example,

  • quay.io

  • mirrors.domino.tech

helm.image_registries.*.username

Docker registry username.

✓

helm.image_registries.*.password

Docker registry password.

✓

Internal Docker registry

The recommended configuration for the internal Docker registry deployed with Domino. Use override values to allow the registry to use S3, GCS, or Azure blob store as a backend store. GCS requires a service account already be bound into the Kubernetes cluster with configuration to ensure the docker-registry service account is properly mapped.

Note
 Either internal_docker_registry or external_docker_registry must be configured.
KeyDescriptionRequiredValues

internal_docker_registry.s3_override.region

AWS region of the S3 bucket store.

For example, us-west-2

internal_docker_registry.s3_override.bucket

S3 bucket name.

For example, domino-bucket-1

internal_docker_registry.s3_override.sse_kms_key_id

KMS Key ARN.

For example, arn:aws:kms:us-west-2:890728157128:key/mrk-eb66487f9d674b46b3ca3c8b934ddd4b

internal_docker_registry.gcs_override.bucket

GCS bucket name.

For example, domino-bucket-1

internal_docker_registry.gcs_override.service_account_name

GCS service account with access to the bucket.

internal_docker_registry.gcs_override.project_name

GCP project name that Domino is deployed into.

internal_docker_registry.azure_blobs_override.account_name

Azure blobstore account name.

internal_docker_registry.azure_blobs_override.account_key

Azure blobstore account key.

internal_docker_registry.azure_blobs_override.container

Azure blobstore container name.

Telemetry

Domino supports user telemetry data to help improve the product.

KeyDescriptionRequiredValues

intercom.enabled

Enable Intercom onboarding.

✓

true, false

mixpanel.enabled

Enable MixPanel.

✓

true, false

mixpanel.redacted_mode

When enabled, telemetry data will be redacted to remove potentially sensitive information.

✓

true, false

Support

Domino supports customer service tools to improve customer experience.

KeyDescriptionRequiredValues

zendesk.enabled

Enable Zendesk.

✓

true, false

zendesk.token

Zendesk API token.

✓

Domino Governance

Configuration for Domino Governance.

KeyDescriptionRequiredValues

ai_governance.enabled

Enable Domino Governance.

✓

true, false

Audit Trail

Domino’s Audit Trail feature captures user-generated events to be queried and viewed in the UI.

KeyDescriptionRequiredValues

audit_trail.enabled

Enable Audit Trail.

✓

true, false

GPU

If using NVIDIA-based GPU compute nodes, enable the following configuration setting to install the required components:

KeyDescriptionRequiredValues

gpu.enabled

Enable GPU support.

✓

true, false

Neuron

If using Neuron Trainium GPU compute nodes, enable the following configuration setting to install the required components:

KeyDescriptionRequiredValues

neuron.enabled

Enable GPU support.

✓

true, false

Fleetcommand

Domino supports minor patch upgrades through an internal tool named Fleetcommand. To learn more about the telemetry being sent back to Domino, see Deployments Telemetry.

KeyDescriptionRequiredValues

fleetcommand.enabled

Enable ability for Domino staff to apply minor patches.

✓

true, false

fleetcommand.url

The URL to fleetcommand, Domino staff will provide this.

fleetcommand.api_token

Deployment-specific API token (Domino staff will provide this).

Node selectors

Domino will by default deploy some DaemonSets on all available nodes in the host cluster. When you run Domino in a multi-tenant Kubernetes cluster, where some nodes must not be used by Domino, you can label nodes for Domino with a single, consistent label. Then, provide that label to fleetcommand-agent with the below configuration to apply a selector to all Domino resources for that label.

KeyDescriptionRequiredValues

global_node_selectors

List of key/value pairs to use as the label for the selector.

Optional

Example:

global_node_selectors:
  domino-owned: "true"

This example applies a selector for domino-owned=true to all Domino deployment resources.

Pod configurations

Global pod configuration that applies to all pods which Domino deploys.

KeyDescriptionRequiredValues

global_pod_annotations

List of key and value pairs to use as annotations that apply to all pods.

Optional

Example:

global_pod_annotation:
  my-annotation: "abc"

This example adds an annotation my-annotation to all Domino pod resources.

global_pod_labels

List of key and value pairs to use as labels that apply to all pods.

Optional

Example:

global_pod_labels:
  my-label: "abc"

This example adds a label my-label to all Domino pod resources.

global_pod_environment

List of name and value pairs to use as environment variables that apply to all pods.

Optional

Example:

global_pod_environment:
  - name: MY_ENV
    value: VALUE

This example adds an environment variable MY_ENV to all Domino pod resources.

Image caching

These settings control the Domino image caching service, which runs as a privileged pod and uses the host Docker socket to pre-pull popular Domino environment images onto compute workers. It can be disabled.

KeyDescriptionRequiredValues

image_caching.enabled

Whether or not to deploy the image caching service.

✓

true, false

Certificate management

KeyDescriptionRequiredValues

certificate_management.install

Whether to install cert-manager. Domino requires cert-manager, but only one instance of cert-manager can be installed in any given Kubernetes cluster. If your Kubernetes cluster already has cert-manager installed, this should be false.

✓

true, false

Teleport Kubernetes agent

No teleport support is installed if teleport_kube_agent is not present.

KeyDescriptionRequiredValues

teleport_kube_agent.proxyAddr

The teleport address.

✓

teleport_kube_agent.authToken

The authentication token for Teleport.

✓

Elasticsearch

KeyDescriptionRequiredValues

elasticsearch.enable_sysctl

To disable the sysctlInitContainer (which runs as root), set the sysctl vm.max_map_count correctly through some other method.

✓

true, false

Image building

A builder will use a storage cache as large as the amount specified by the cache_storage_size. When a build completes, the storage cache is reduced to the cache_storage_retention amount. The cache_storage_retention value must not be greater than the cache_storage_size.

When the underlying OS does not support user namespace mapping, like EKS, you might have to disable rootless building for deployment targets. Running rootless in an environment that does not support it defaults to using the native filesystem snapshotter and causes image building performance to drop significantly.

KeyDescriptionRequiredValues

image_building.verbose

Enable verbose logging.

✓

true, false

image_building.rootless

Run as a non-root user.

✓

true, false

image_building.concurrency

Maximum number of concurrent image builds.

image_building.cache_storage_size

Size of each image builder’s cache.

image_building.cache_storage_retention

Amount of storage to keep during garbage collection.

image_building.builder_node_selector

List of key/value pairs to use as the label for the selector.

image_building.pool_sync_wait_time

Duration after which builder cluster is inspected for idle pods.

Examples: 1h, 5m, 30s

image_building.pool_max_idle_time

Duration after which idle build pods are terminated.

Examples: 1h, 5m, 30s

image_building.fetch_and_extract_timeout

Duration the build waits to fetch and extract the remote Docker context.

Examples: 1h, 5m, 30s

image_building.pool_endpoint_watch_timeout

Number of seconds the worker pool waits for a buildkit pod to become ready for traffic.

Registries

Configures buildkit/hephaestus to support HTTP-only and/or self-signed registries.

KeyDescriptionRequiredValues

image_building.registries.*.http

Registry only supports HTTP.

✓

true, false

image_building.registries.*.insecure

Registry uses a self-signed certificate and is considered insecure.

✓

true, false

Cloud registry authentication

Set up cloud authentication so that the image builder can pull images from and push images to cloud container registries.

Azure

KeyDescriptionRequiredValues

image_building.cloud_registry_auth.azure.tenant_id

Azure AD Directory (tenant) ID.

✓

image_building.cloud_registry_auth.azure.client_id

Azure AD Application (client) ID.

✓

image_building.cloud_registry_auth.azure.client_secret

Azure AD Application secret, required when using a Service Principal.

image_building.cloud_registry_auth.azure.workload_identity

Azure AD Application (client) ID represents a Workload Identity.

✓

true, false

Google Cloud Platform (GCP)

KeyDescriptionRequiredValues

image_building.cloud_registry_auth.gcp.service_account

Service account with access to target GCR/GAR registry.

✓

Service host prefixes

Certain services (currently only apps) can be configured with a separate subdomain.

KeyDescriptionRequired

service_host_prefixes.app

Optional subdomain prefix for apps.

Admin Toolkit

The Domino Admin Toolkit compiles troubleshooting data for debugging and support.

KeyDescriptionRequired

admin_toolkit.enable_report_uploads

Upload toolkit report to Domino.

true, false

FIPS

Control whether Domino is FIPS 140-2 compliant.

KeyDescriptionRequiredValues

fips

Enables a FIPS compliant configuration.

Optional

true, false

Note

Enabling FIPS compliance will require the following:

  1. The Kubernetes node image must have FIPS enabled.

  2. To ensure compliance with the fips-140-2 policy, a FIPS enabled version of Istio must be installed and enabled. This can be achieved by setting fips=true and istio.install=true. Otherwise, if you’re providing your own version of Istio, it must be FIPS compliant.

  3. For EKS, the following must be added to global_pod_environment.

global_pod_environment:
  - name: AWS_USE_FIPS_ENDPOINT
    value: "true"

Domino Flows

Domino Flows enables efficient orchestration and monitoring of multi-step processes while ensuring full lineage and reliable reproducibility. Domino Flows is implemented on top of Flyte, which is an open-source task orchestrator that facilitates building production-grade data and ML pipelines. The orchestrator stores task data (inputs and outputs) and task metadata in blob storage. Configure the following so that Domino Flows has appropriate access to blob storage.

AWS

KeyDescriptionRequiredValues

flyte.metadata_storage.s3.region

AWS region of the Flyte metadata S3 bucket.

For example, us-west-2

flyte.metadata_storage.s3.bucket

Flyte metadata S3 bucket name.

For example, domino-flyte-metadata-bucket-1

flyte.metadata_storage.s3.sse_kms_key_id

KMS key ID associated with the Flyte metadata S3 bucket.

For example, 2f38143f-a42e-4f11-9ebb-162f11a6c350

flyte.data_storage.s3.region

AWS region of the Flyte data S3 bucket.

For example, us-west-2

flyte.data_storage.s3.bucket

Flyte data S3 bucket name.

For example, domino-flyte-data-bucket-1

flyte.data_storage.s3.sse_kms_key_id

KMS key ID associated with the Flyte data S3 bucket.

For example, 2f38143f-a42e-4f11-9ebb-162f11a6c350

flyte.cluster_config.eks.aws_account_id

AWS account ID associated with the Flyte S3 buckets.

For example, 012345678901

flyte.cluster_config.eks.controlplane_role_arn

ARN for the AWS role that has access to the Flyte metadata S3 bucket.

For example, arn:aws:iam::012345678901:role/domino-flyte-controlplane

flyte.cluster_config.eks.dataplane_role_arn

ARN for the AWS role that has access to the Flyte data S3 bucket.

For example, arn:aws:iam::012345678901:role/domino-flyte-dataplane

Azure

KeyDescriptionRequiredValues

flyte.metadata_storage.azure.account_name

Azure storage account name for the Flyte metadata container.

For example, domino-flyte-account-name

flyte.metadata_storage.azure.account_key

Access key for the Flyte metadata storage account.

For example, [to come]

flyte.metadata_storage.azure.container

Name of the Flyte metadata container in the storage account.

For example, domino-flyte-metadata-container

flyte.data_storage.azure.account_name

Azure storage account name for the Flyte data container.

For example, [to come]

flyte.data_storage.azure.account_key

Access key for the Flyte data storage account.

For example, [to come]

flyte.data_storage.azure.container

Name of the Flyte data container in the storage account.

For example, domino-flyte-data-container

flyte.cluster_config.aks.tenant_id

ID of the tenant associated with the storage containers.

For example, fc04e151-a39c-40a1-96d9-3913aa8a4a46

flyte.cluster_config.aks.subscription_id

Azure subscription ID associated with the storage containers.

For example, f914720b-d639-4832-bc9c-4d6f2cdebce4

flyte.cluster_config.aks.controlplane_client_id

Azure client ID associated with the metadata storage.

For example, 3831e719-05b8-470f-a171-a3b34ca50af9

flyte.cluster_config.aks.dataplane_client_id

Azure client ID associated with the data storage.

For example, 01770221-a803-49fb-bda4-94033a87638e