domino logo
About DominoArchitecture
Kubernetes
Cluster RequirementsDomino on EKSDomino Kubernetes Version CompatibilityDomino on GKEDomino on AKSDomino on OpenShiftNVIDIA DGX in DominoDomino in Multi-Tenant Kubernetes ClusterEncryption in Transit
Installation
Installation ProcessConfiguration ReferenceInstaller Configuration ExamplesPrivate or Offline InstallationCustom Certificatesfleetcommand-agent Release NotesInstall Script Downloads
Azure Deployments
Prepare to InstallProvision Infrastructure and Runtime EnvironmentDeploy DominoKubernetes Upgrade Guide
Google Cloud Deployments
Prepare to InstallProvision Infrastructure and Runtime EnvironmentDeploy DominoKubernetes Upgrade Guide
Amazon Web Services Deployments
Prepare to InstallProvision Infrastructure and Runtime EnvironmentDeploy DominoKubernetes Upgrade Guide
Configuration
Central ConfigurationNotificationsFeature FlagsChange The Default Project For New UsersProject Stage ConfigurationDomino Integration With Atlassian Jira
Compute
Manage Domino Compute ResourcesHardware Tier Best PracticesModel Resource QuotasPersistent Volume ManagementAdding a Node Pool to your Domino ClusterRemove a Node from Service
Keycloak Authentication Service
Operations
Domino Application LoggingDomino MonitoringSizing Infrastructure for Domino
Data Management
Data in DominoData Flow In DominoExternal Data VolumesDatasets AdministrationSubmit GDPR Requests
User Management
RolesManage UsersView User InformationRun a User Activity ReportSchedule a User Activity Report
Environments
Environment Management Best PracticesCache Environment Images in EKSImages From Authenticated External Registries
Backup and Restore
Backup StructureBackup LocationCustomize BackupsRun a Manual, On-Demand BackupRestore backups
Control Center
Control Center OverviewExport Control Center Data with The API
Troubleshooting
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
Admin Guide
>
Installation
>
Installation Process

Installation Process

The Domino platform runs on Kubernetes. To simplify deployment and configuration of Domino services, Domino provides an install automation tool called the fleetcommand-agent that uses Helm to deploy Domino into your compatible cluster. The fleetcommand-agent is a Python application delivered in a Docker container, and can be run locally or as a job inside the target cluster.

install diagram

Requirements

The install automation tools are delivered as a Docker image, and must run on an installation workstation that meets the following requirements:

  • Docker installed.

  • Kubectl service account access to the cluster.

  • Access to download and install Helm through package manager or GitHub.

  • Access to quay.io and credentials for an installation service account with access to the Domino installer image and upstream image repositories. Throughout these instructions, these credentials will be referred to as $QUAY_USERNAME and $QUAY_PASSWORD. Contact your Domino account team if you need new credentials.

The fleetcommand-agent needs access to the following types of assets to install Domino:

  1. Docker images for Domino components

  2. Helm charts

The hosting cluster must have access to the following domains through Internet to retrieve component and dependency images for online installation:

  • quay.io

  • domino.tech

  • k8s.gcr.io

  • docker.elastic.co

  • docker.io

  • gcr.io

Alternatively, you can configure the fleetcommand-agent to point to a private docker registry and application registry for offline installation. Offline installation packages are downloaded from the mirrors.domino.tech domain.

Domino recommends that you use a partition with at least ten times more disk space than the total volume of your container images. For example, if your container images take up several dozen gigabytes, use a partition with several hundred gigabytes of space available. This ensures that the partition has enough free space for container images that Domino users create and manage. See Why did my execution fail with the error 'No space left on device'? for advice if you exceed your available disk space.

Pull the fleetcommand-agent image

  1. Log in to quay.io with the credentials described in the requirements previous section.

    docker login quay.io
  2. Find the image URI for the version of the fleetcommand-agent you want to use from the release notes.

  3. Pull the image to your local machine.

    docker pull quay.io/domino/fleetcommand-agent:v54

Run fleetcommand-agent commands

The default entrypoint for the fleetcommand-agent is:

"Entrypoint": [ "python", "-m", "fleetcommand_agent"]

This launches the Python application inside the container at /app/fleetcommand_agent. This allows you to easily run agent commands through docker run like this:

docker run --rm quay.io/domino/fleetcommand-agent:v54 $COMMAND $ARGUMENTS

The fleetcommand-agent supports the following commands:

init

Generates a template configuration file.

Arguments:

--file or -f

File system path to write the template to. This must be a host volume mounted to the container to persist the output.

--full or -F

Includes optional and advanced portions of the template. Should only be used when advanced options are needed, as configurations with these fields are more complex to maintain.

--version

Domino version to generate a configuration template for.

--image-registry

Provide a registry URI to prepend to Domino images to set up the template for installation from a private Docker registry. Should be used in conjunction with --full.

Example

docker run --rm -v $(pwd):/install quay.io/domino/fleetcommand-agent:v54 \
init --file /install/domino.yml

run

Installs Domino into a cluster specified by a Kubernetes configuration from the KUBECONFIG environment variable. A valid configuration file must be passed in to this command.

Arguments:

--file or -f

File system path to the complete and valid configuration file.

--kubeconfig

Path to Kubernetes configuration file containing cluster and authentication information to use.

--dry

Use this mode to not make any permanent changes to the target cluster. A dry run checks service account permissions and generates detailed logs about the charts to be deployed with the given configuration. The output is written to \'/app/logs and /app/.appr_chart_cache inside the container.

This option requires that the namespaces you want to use already exist.

Example:

docker run --rm -v $(pwd):/install quay.io/domino/fleetcommand-agent:v54 run --file /install/domino.yml

Example dry run:

docker run --rm -v $(pwd):/install -v $(pwd)/logs:/app/logs \
-v $(pwd)/cache:/app/.appr_chart_cache quay.io/domino/fleetcommand-agent:v54 \
run --dry --file /install/domino.yml

destroy

Removes all resources from the target cluster for a given configuration file.

Arguments:

--file or -f

File system path to the complete and valid configuration file.

--kubeconfig

Path to Kubernetes configuration file containing cluster and authentication information to use.

--dry

Use this mode to not make any permanent changes to the target cluster. A dry run checks service account permissions and generates detailed logs about the charts to be deployed with the given configuration.

Example:

docker run --rm -v $(pwd):/install quay.io/domino/fleetcommand-agent:v54 \
destroy --file /install/domino.yml

Install process

  1. Connect to a workstation that meets the installation automation requirements listed previously. You do not have to use an account with root privileges to run the installer.

  2. Log in to quay.io with the credentials described in the previous requirements section.

    docker login quay.io
  3. Retrieve the Domino installer image from quay.io.

    docker pull quay.io/domino/fleetcommand-agent:v54
  4. Initialize the installer application to generate a template configuration file named domino.yml.

    docker run --rm -it -v $(pwd):/install quay.io/domino/fleetcommand-agent:v54 \
    init --file /install/domino.yml
  5. Edit the configuration file with the details about the target cluster, storage systems, and hosting domain. Read the configuration reference for more information about available keys, and consult the configuration examples for guidance about getting started.

    Change the value of name from domino-deployment to something that identifies the purpose of your installation and contains the name of your organization.

  6. Run this install script from the directory with the finalized configuration file to install Domino into the cluster. Fill in your $QUAY_USERNAME and $QUAY_PASSWORD where indicated, and also this script assumes your installer configuration file is in the same directory, and is named exactly domino.yml.

    #!/bin/bash
    
    set -ex
    
    kubectl delete po --ignore-not-found=true fleetcommand-agent-install
    
    kubectl create secret \
      docker-registry \
      -o yaml --dry-run \
      --docker-server=quay.io \
      --docker-username=$QUAY_USERNAME \
      --docker-password=$QUAY_PASSWORD \
      --docker-email=. domino-quay-repos | kubectl apply -f -
    
    kubectl create configmap \
      fleetcommand-agent-config \
      -o yaml --dry-run \
      --from-file=domino.yml | kubectl apply -f -
    
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: admin
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: admin-default
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: admin
      namespace: default
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: fleetcommand-agent-install
    spec:
      serviceAccountName: admin
      imagePullSecrets:
        - name: domino-quay-repos
      restartPolicy: Never
      containers:
      - name: fleetcommand-agent
        image: quay.io/domino/fleetcommand-agent:v54
        args: ["run", "-f", "/app/install/domino.yml", "-v"]
        imagePullPolicy: Always
        volumeMounts:
        - name: install-config
          mountPath: /app/install/
      volumes:
      - name: install-config
        configMap:
          name: fleetcommand-agent-config
    EOF
    
    set +e
    while true; do
      sleep 5
      if kubectl logs -f fleetcommand-agent-install; then
        break
      fi
    done
  7. The installation process can take up to 30 minutes to fully complete. The installer will output verbose logs and surface any errors it encounters, but it can also be useful to follow along in another terminal tab by running:

    kubectl get pods --all-namespaces

    This will show the status of all pods being created by the installation process. If you see any pods enter a crash loop or hang in a non-ready state, you can get logs from that pod by running:

    kubectl logs $POD_NAME --namespace $NAMESPACE_NAME

    If the installation completes successfully, a message opens that says:

    <timestamp> INFO - fleetcommand_agent.Application - Deployment complete.
    Domino is accessible at $YOUR_FQDN

    However, the application will only be accessible through HTTPS at that FQDN if you have configured DNS for the name to point to an ingress load balancer with the appropriate SSL certificate that forwards traffic to your platform nodes.

Upgrade

Upgrading a Domino deployment is a simple process of running the installer again with the same configuration, but with the version field set the value of the desired upgrade version. See the Configuration Reference and the installer release notes for information on the Domino versions your installer can support.

If you must upgrade to a newer installer version to upgrade to a Domino version, use the following process.

  1. Retrieve the new Domino installer image from quay.io by filling in the version value in the following command.

    docker pull quay.io/domino/fleetcommand-agent:v54
  2. Create a copy of your existing domino.yml configuration file. Name it something like domino-orig-<date>.yaml, indicating it is the version of the configuration before modifications were made on a specific date.

  3. Generate a new domino.yml configuration template by running the initialization command through the new version of the installer. This will ensure you have a configuration schema that conforms to the new version.

    docker run --rm -it -v $(pwd):/install quay.io/domino/fleetcommand-agent:v54 \
      init --file /install/domino.yml
  4. Copy custom values from your original configuration into the new file.

  5. When complete, run the install script from the install process and change the spec.containers.image value to `quay.io/domino/fleetcommand-agent:v54 (with the appropriate version).

Test and troubleshoot

Run the following tests to verify that your Domino installation was successful:

  • Login to the Domino application and that all the navigation panel options are operational.

    Failure of this test means you must check that Keycloak was setup properly.

  • Create a new project and launch a Jupyter/JupyterLab workspace.

    Failure of this test mean you must check that default environment images have been loaded in the cluster.

  • Verify that the Control Center page is operational.

    Failure of this test means you must check that all the pods in the domino-deployment namespace are running correctly.

  • Verify that the Dispatcher page is operational.

    Failure of this test means you must check that all the pods in the domino-deployment namespace are running correctly.

  • Publish an app with flask or shiny apps.

    Failure of this test means you must check that the environment images have flask and shiny installed.

  • Verify that email notifications are operational by completing a run and verifying that an email confirmation has been sent to you.

    Failure of this test means you must verify there are valid email addresses for users in the admin section.

  • VLaunch a new model (confirm through the API tester and curl it).

    Failure of this test means you must check that Domino can create new objects in the cluster.

  • Verify that datasets work by starting a workspace and adding a fie to the domino/datasets/<username>/<project_name>/scratch directory and seeing that it shows up in Domino after closing the workspace.

    Failure of this test means you must check your data and storage configurations in the admin section.

Enable user registration

Use Keycloak to enable user registration, so users can access your fresh Domino install. Keycloak is a user authentication service that runs on a pod in your cluster.

  1. Sign in to Keycloak on your Domino instance.

  2. In the Keycloak sidebar menu, select Realm Settings.

  3. Select the Login tab, and toggle User registration to On.

  4. Click Save to confirm your changes.

Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.