The following are the types of logs produced by the operation of Domino.
-
Domino execution logs
-
Domino application logs
These are the logs output by user code running in Domino as a Job, Workspace, App, or Model API. These are available in the Domino web application on the Jobs dashboard, Workspaces Dashboard, App Dashboard, and Model API instance logs. This data is a key part of the Domino reproducibility model, and is kept indefinitely in the Domino blob store.
blob_storage.logs.
All Domino services output their logs using the standard Kubernetes logging architecture. Relevant logs are printed to stdout or stderr as indicated, and are captured by Kubernetes.
For example, to look at your front end logs you can do the following:
-
List your all namespaces to find the name of you platform namespace:
kubectl get namespace -
List all the pods in your platform namespace to find the name of a frontend. You will likely have more than one frontend pod.
kubectl get pods -n <namespace for your platform nodes> -
Print the frontend logs for one of your frontends:
kubectl logs <pod name of your frontend pod> -n <namespace for your platform nodes> -c nucleus-frontend
The most effective way to aggregate logs is to attach a Kubernetes log aggregation utility to monitor the following Kubernetes namespaces used by Domino:
- Platform namespace
This namespace hosts the core application components of the Domino
application, including API servers, databases, and web interfaces. The
name of this namespace is defined in the installer configuration file at
namespaces.platform.name.
The following components running in this namespace produce the most
important logs:
| Component | Logs |
|---|---|
| The nucleus-frontend pods host the frontend API server that routes all requests to the Domino application. Its logs will contain details on HTTP requests to Domino from the application or another API client. If you see errors in Domino with HTTP error codes like 500, 504, or 401, you can find corresponding logs here. |
| The nucleus-dispatcher pod hosts the Domino scheduling and brokering service that sends user execution pods to Kubernetes for deployment. Errors in communication between Domino and Kubernetes will result in corresponding logs from this service. |
| The keycloak pods hosts the Domino authentication service. The logs for this service will contain a record of authentication events, including additional details on any errors. |
| This pod hosts the open-source Kubernetes cluster autoscaler, which controls and manages autoscaling resources. The logs for this service will contain records of scaling events, both scaling up new nodes in response to demand and scaling down idle resources, including additional details on any errors. |
- Compute grid namespace
This namespace hosts user executions plus Domino environment builds. The
name of this namespace is defined in the installer configuration file at
namespaces.compute.name.
Logs in this namespace correspond to ephemeral pods hosting user work. Each pod will contain a user-defined environment container, whose logs are described previously as Execution logs. There are additional supporting containers in those pods, and their logs might contain additional information on any errors or behavior seen with specific Domino executions.
Domino advises that you aggregate and keep at least 30 days of logs to facilitate debugging. These logs can be harvested with a variety of Kubernetes log aggregation utilities, including:
You can enable audit logging for several events. Audit logging for models has been improved in the 4.6.1 release. These are the major model events that are logged when triggered through the Domino UI:
-
New model create
-
New model version publish
-
Model version stop / start
-
Model archived
-
Model collaborator add / change / remove
-
Model settings change
-
Syslog server
-
Mixpanel
After you enable audit logging, messages are written to Application logs. Other log targets require additional configuration.
Contact support@dominodatalab.com for assistance enabling, accessing, and processing audit logs.
Details for audit logging of data interactions can be found in Tracking and auditing data interactions in Domino.