domino logo
About DominoArchitecture
Kubernetes
Cluster RequirementsDomino on EKSDomino Kubernetes Version CompatibilityDomino on GKEDomino on AKSDomino on OpenShiftNVIDIA DGX in DominoDomino in Multi-Tenant Kubernetes ClusterEncryption in Transit
Installation
Installation ProcessConfiguration ReferenceInstaller Configuration ExamplesPrivate or Offline InstallationCustom Certificatesfleetcommand-agent release notes
Azure Deployments
Prepare for InstallationDeploy DominoProvision Infrastructure and Runtime Environment
Google Cloud Deployments
Prepare for InstallationProvision Infrastructure and Runtime EnvironmentDeploy Domino
Configuration
Central ConfigurationNotificationsFeature FlagsChange The Default Project For New UsersProject Stage ConfigurationDomino Integration With Atlassian Jira
Compute
Manage Domino Compute ResourcesHardware Tier Best PracticesModel Resource QuotasPersistent Volume ManagementAdding a Node Pool to your Domino ClusterRemove a Node from Service
Keycloak Authentication Service
Operations
Domino Application LoggingDomino MonitoringSizing Infrastructure for Domino
Data Management
Data in DominoData Flow In DominoExternal Data VolumesDatasets AdministrationSubmit GDPR Requests
User Management
RolesManage UsersLicense Usage Reporting
Environments
Environment Management Best PracticesCache Environment Images in EKSImages From Authenticated External Registries
Backup and Restore
Backup StructureBackup LocationCustomize BackupsRun a Manual, On-Demand BackupRestore backups
Control Center
Control Center OverviewExport Control Center Data with The API
Troubleshooting
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
Admin Guide
>
Troubleshooting

Troubleshooting

Use the Administrator toolkit to run a system scan on your Domino deployment to do the following:

  • Verify its health.

  • Get information to assist troubleshooting.

  • Get assistance to run remediation actions.

This toolkit can be used on Domino 4.x and higher.

Prerequisites

  • Bastion host, laptop, or other host with kubectl access that has network connectivity to the target cluster

  • Use the following command to get the latest version of toolkit.sh:

    curl -sSO https://domino-admin-toolkit.s3-us-west-2.amazonaws.com/toolkit.sh && chmod a+x ./toolkit.sh

Run the toolkit

  1. Run the toolkit. This outputs a report in the console and uploads it to a location for Domino staff.

    ./toolkit.sh
    1. If the Domino installation blocks AWS access, run the following command to output the report only in the console:

      Note
      ./toolkit.sh up
      ./toolkit.sh pytest
    2. If you do not have access to quay.io and mirrors.domino.tech, push the toolkit’s docker image to your internal registry.

      1. Run the following from the host where you have internet and internal docker registry access:

        ./toolkit.sh push internal-registry-host/domino-admin-toolkit
      2. From wherever you have kubectl access or from central, run the following:

        ./toolkit.sh up --image internal-registry-host/domino-admin-toolkit
        ./toolkit.sh pytest

View the reports

Use the Administrator’s toolkit web application to review the reports in HTML format. This application automatically terminates after one hour.

  1. From wherever you have kubectl access, run the following:

    ./toolkit.sh up --load-balancer

    The output is similar to the following:

    Pod is Running ok!
    Web UI: http://load-balancer-id.us-west-2.elb.amazonaws.com:8888/?exclude=salt

    cli report

  2. Go to the link specified in the Web application output.

    web report

./toolkit.sh usage

The following describes the toolkit parameters.

./toolkit.sh help

Usage: toolkit.sh [command] [options]

Run with no parameters to create resources and run the tests

Commands:
  up     [options]  Create resources but don't run the tests
  test   [options]  Run tests with options and upload HTML report directly to Domino
  pytest [options]  Run pytest directly with given options
  clean             Delete resources. Best practice is to delete after you are done.
  exec [command]    Execute a command in the toolkit container (by default /bin/bash). Useful for debugging.
  logs              Show toolkit container logs
  help              Get this help message

`up` command options:
  ./toolkit.sh up --tag|-t tag     Run specific docker image tag of the admin toolkit
  ./toolkit.sh up --load-balancer  Expose admin toolkit using the load balancer, by default it is exposed as a node port

`test` command options:
  ./toolkit.sh test --help                  Show help including choice of parameters
  ./toolkit.sh test --exclude salt kubeadm  Exclude tests for specific features
  ./toolkit.sh test --log-cli-level DEBUG   Show extra debugging output
Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.