User code running as Jobs, Workspaces, Apps, and Domino endpoints output execution logs.
Your users can access these from the Jobs dashboard, Workspaces Dashboard, App Dashboard, and Domino endpoint instance logs. This data is a key part of the Domino reproducibility model and is kept indefinitely in the Domino File Store.
The installation configuration file at blob_storage.logs
defines which system these logs are written to.
All Domino services use the standard Kubernetes logging architecture to output their logs.
Relevant logs are printed to stdout
or stderr
as indicated, and Kubernetes captures them.
A zip file that contains logs and reports from Domino components with information about a Domino execution. See Support Bundles.
-
List your all namespaces to find the name of your platform namespace:
kubectl get namespace
-
List all the pods in your platform namespace to find the name of a front-end. You will likely have more than one frontend pod.
kubectl get pods -n <namespace for your platform nodes>
-
Print the front-end logs for one of your frontends.
kubectl logs <pod name of your frontend pod> -n <namespace for your platform nodes> -c nucleus-frontend
Domino recommends that you attach a Kubernetes log aggregation utility to monitor the following Kubernetes namespaces used by Domino to aggregate the logs.
-
Platform namespace
This namespace hosts the core application components of the Domino application, including API servers, databases, and web interfaces. The installer configuration file defines the name of this namespace at
namespaces.platform.name
. The following components running in this namespace produce the most important logs:Component Logs nucleus-frontend
The nucleus-frontend pods host the frontend API server that routes all requests to the Domino application. Its logs will contain details about HTTP requests to Domino from the application or another API client. If you see errors in Domino with HTTP error codes like 500, 504, or 401, you can find corresponding logs here.
nucleus-dispatcher
The nucleus-dispatcher pod hosts the Domino scheduling and brokering service that sends user execution pods to Kubernetes for deployment. Errors in communication between Domino and Kubernetes result in corresponding logs from this service.
keycloak
The keycloak pod hosts the Domino authentication service. The logs for this service will contain a record of authentication events, including additional details on any errors.
cluster-autoscaler
This pod hosts the open-source Kubernetes cluster autoscaler, which controls and manages autoscaling resources. The logs for this service contain records of scaling events, both scaling up new nodes in response to demand and scaling down idle resources, including additional details on any errors.
-
Compute grid namespace
This namespace hosts user executions plus Domino environment builds. The installer configuration file defines the name of this namespace at
namespaces.compute.name
.Logs in this namespace correspond to ephemeral pods hosting user work. Each pod contains a user-defined environment container, whose logs are execution logs. There are additional supporting containers in the pods, and their logs might contain additional information about errors or behavior seen in specific Domino executions.
Domino advises that you aggregate and keep at least 30 days of logs to facilitate debugging. You can harvest these with a variety of Kubernetes log aggregation utilities, including:
-
Project audit logs for project-level auditing including user permissions and Dataset-related actions.