Run the admin toolkit

The admin toolkit is an always-on tool deployed alongside Domino in the Kubernetes cluster that allows Domino administrators to run a system scan on the Domino deployment to do the following:

  • Check for unpatched known issues

  • Validate the health of core services

  • Get information to assist troubleshooting, run remediation actions, or provide context about issues affecting the deployment

  • Provide the toolkit report to Domino Support as part of an incident

The toolkit has a front-end web interface provided at https://<your-domino-url>/toolkit/ where you can view scan reports and initiate new scans. There is also a CLI tool to initiate scans and help manage the toolkit deployment.

You can run scans as many times as necessary because the checks performed by the toolkit are only reading information and not making any modifications. You can run a scan during any of these stages:

  • After a new Domino install

  • At the start of an incident

  • In the midst of an incident

  • After incident resolution to verify health

  • Before and after a Domino upgrade

  • Before and after a Domino migration

You can find details about the specific tests that are included in the scan in the Admin toolkit documentation.

You can find check runbooks at our Support site knowledge base (requires sign-in).

Access Admin Toolkit

Admin Toolkit is available at the URL https://<your-domino-domain>/toolkit/ or through the Domino Admin UI under Advanced > Diagnostics.

Out of the box, Admin Toolkit should be configured with SSO so that users will automatically be logged in to the Toolkit UI if they are logged in to Domino. If they are not currently logged in to Domino or their session has expired, then they will see the Admin Toolkit login screen.

Clicking on Continue with Domino SSO will take the user to the normal Domino login screen where they can log in using their Domino credentials as normal. Once logged in, Domino will redirect back to the Toolkit UI.

All Domino SysAdmin users are automatically granted the Toolkit Admin role.

Login when SSO is unavailable

When SSO is not working (for example, Keycloak is unavailable for some reason), then there is a backup method to access Toolkit with Admin privileges.

Get the Admin Toolkit password

When the toolkit is installed, the installer generates a random password that does not change for the life of the toolkit deployment. If you installed the toolkit on the command line using the CLI, this password (along with the ingress URL) is displayed at the end of the installation process.

If you forget the password, or if toolkit was installed by the Domino installer, you can retrieve the password using this command:

./toolkit.sh get-password

Navigate to the Toolkit login page and do the following:

  1. Enter the login you retrieved above.

  2. Enter the password you retrieved above.

  3. Click Login.

Get the Admin Toolkit password without the toolkit.sh CLI

The Admin Toolkit password is stored in a secret called domino-admin-toolkit-http in the domino-platform namespace. You can also retrieve the password with kubectl:

kubectl get secret -n domino-platform domino-admin-toolkit-http -o jsonpath='{.data.webui-password}' | base64 -d; echo

The default username is admin-toolkit.

Run a system scan

You can run a system scan using the Admin Toolkit web interface or CLI.

Run a scan using the web interface

  1. Navigate to https://<your-domino-url>/toolkit/ and log in.

  2. Click the Run New Report button at the bottom of the page:

    Toolkit user interface

    When the scan has finished, the web interface displays a link to the report.

Run a scan using the CLI

From your bastion host or laptop where you have toolkit.sh saved, run this command:

./toolkit.sh test

This starts a scan, displaying the progress to the console. The completed scan is available as HTML in the web interface.

Run a scan on an air-gapped deployment

If your Domino deployment blocks AWS access, run the following command to output the report only in the console:

./toolkit.sh pytest

This runs the report and displays it to the console with no colorizing of the output, making it easier to redirect the output to a file or copy and paste from the console into a text file that can be sent to Domino. Contact Domino support at support@dominodatalab.com to pre-arrange a secure way to send the text file to Domino.

Troubleshooting

Attempting to run ./toolkit.sh results in the following error:

/usr/local/bin/python: Error while finding module specification for 'domino_admin_toolkit.test_runner' (ModuleNotFoundError: No module named 'domino_admin_toolkit')
error: Internal error occurred: error executing command in container: Internal error occurred: error executing command in container: command terminated with exit code 1

This indicates there is still a v1 toolkit installation present. Run ./toolkit.sh uninstall; ./toolkit.sh install to remove it and install the latest version.

View the reports

Use the Admin Toolkit’s web interface to review the reports in HTML format. The web interface displays a list of the 10 most recent reports that have been generated by the toolkit. Click the link of the report you wish to view.

Toolkit web report
Note
 Reports are stored on the toolkit’s pod storage and are not persisted across pod restarts. With kubectl access you can download all stored reports using: kubectl cp -n domino-platform -c domino-admin-toolkit <pod_name>:/app/domino_admin_toolkit/ui/templates/reports/ ./reports/ replacing <pod_name> with the complete name of the toolkit pod.

Send reports to Domino

Regardless of whether a scan runs from the web interface or CLI, the toolkit can automatically and securely upload the report to a secure centralized S3 bucket owned by Domino within Domino’s internal AWS account. Access to the reports is restricted to authorized Domino engineers through a secure single sign-on (SSO) system with granted permissions.

This behavior is on by default, but can be disabled by either selecting the option in the web interface to opt out of uploads, applying a ConfigMap, or when using the command line by running with the --local-only option:

./toolkit.sh test --local-only
Note
 The --local-only option applies to that run only and does not change the default behavior.

If you have opted out of uploads, you can still upload a report to Domino using the CLI:

./toolkit.sh test --upload-report

There are many benefits to uploading reports to Domino:

  • Quicker resolution of support tickets.

  • Validation of deployment configuration.

  • Receive more proactive, rather than reactive, support.

Domino can also use the data collected from multiple customers to better understand how customer-specific configuration and infrastructure affect the overall performance and stability of the Domino platform so that we can deliver a more robust and performant product with each release.

Important
 Admin Toolkit reports do contain potentially sensitive information such as IP addresses, hostnames, partial log files and usernames. Passwords are not included. If you have concerns about any of the information collected, contact us at support@dominodatalab.com or through your CSM or TAM to discuss your concerns. Since the always-on Admin Toolkit can be updated independently of the Domino version of your deployment, addressing such concerns can be quick.

Enable/disable reports to be sent to Domino

By default, the functionality to have reports uploaded to Domino is enabled.

Each newly generated toolkit report is securely sent to Domino Support and Engineering. This allows Domino engineers to gain access to the reports faster during an outage and get historical context to previous outages. You can opt out of this service (or opt in again) via the web interface or through an applied Kubernetes ConfigMap.

To opt out of sending reports to Domino via the web interface, navigate to the Settings tab and then click the Opt Out of sending reports to Domino button. To opt back in, click the Opt Into sending reports to Domino button.

To opt out of sending reports to Domino through an applied Kubernetes ConfigMap, follow the instructions below.

Note
 These reports only collect deployment and service verification data and do not contain any sensitive data.

ConfigMap

A ConfigMap is generated by default. Here is an example of the default ConfigMap:

apiVersion: v1
name: domino-admin-toolkit-config
data:
  crontab_entry: '* * * * *'
  send_to_domino: "True"
kind: ConfigMap

To opt out of sending reports to Domino, change the send_to_domino value to "False".

Scheduled reports

Scheduled reporting lets you generate Domino health reports at regular intervals. Having a "healthy state" report handy provides a comparative baseline during system disruptions. This can be invaluable in swiftly identifying system anomalies, thus accelerating the issue resolution process.

Set up scheduled reporting

Follow the steps below to set up scheduled reporting:

  1. Open the toolkit web interface and navigate to the Settings tab.

  2. Under the Schedule Admin Toolkit section, specify the frequency and timing for the reports as per your requirement. This is accepted in cron formatting only.

  3. Click Submit.

Toolkit scheduled reports

Now the toolkit generates and stores system health reports according to the schedule you’ve set. These reports can be accessed and reviewed anytime, providing you with a consistent and regular overview of your system’s health.

Important

By default:

  • The schedule is set to once a day.

  • A maximum number of 10 reports are saved at any one time.

Utilize scheduled reports during system disruptions

In case of a system disruption or outage:

  1. Access the most recent "healthy state" report from the stored scheduled reports.

  2. Generate an immediate system health report (if possible).

  3. Compare the two reports to identify any deviations or unusual activities.

Sharing this information with the Domino support team can expedite the troubleshooting process and guide us toward the root cause of the problem more efficiently.

By leveraging scheduled reporting, you can manage system health more effectively, contributing to smoother, more reliable operations in your Domino environment.

Understand and resolve failures

The Admin Toolkit documentation provides comprehensive information about the checks, their meanings, and any details on resolving issues. Use the toolkit and its documentation to quickly identify and resolve issues, and ensure optimal performance of the Domino platform. You can access the documentation in two ways:

  • Publicly-accessible online documentation:

  • Documentation included within the Domino deployment: https://<your-domino-url>/toolkit/docs/ (specific to the deployed version of the toolkit).