Application execution logs

Execution logs

User code running as Jobs, Workspaces, Apps, and Model APIs output execution logs.

Your users can access these from the Jobs dashboard, Workspaces Dashboard, App Dashboard, and Model API instance logs. This data is a key part of the Domino reproducibility model and is kept indefinitely in the Domino File Store.

The installation configuration file at blob_storage.logs defines which system these logs are written to.

Application logs

All Domino services use the standard Kubernetes logging architecture to output their logs. Relevant logs are printed to stdout or stderr as indicated, and Kubernetes captures them.

Support bundle

A zip file that contains logs and reports from Domino components with information about a Domino execution. See Support Bundles.

View your front-end logs

List your all namespaces to find the name of your platform namespace:
```
kubectl get namespace
```
List all the pods in your platform namespace to find the name of a front-end. You will likely have more than one frontend pod.
```
kubectl get pods  -n <namespace for your platform nodes>
```

Print the front-end logs for one of your frontends.

kubectl logs <pod name of your frontend pod> -n <namespace for your platform nodes> -c nucleus-frontend

Domino recommends that you attach a Kubernetes log aggregation utility to monitor the following Kubernetes namespaces used by Domino to aggregate the logs.

Platform namespace

This namespace hosts the core application components of the Domino application, including API servers, databases, and web interfaces. The installer configuration file defines the name of this namespace at namespaces.platform.name. The following components running in this namespace produce the most important logs:

Component Logs

Component	Logs
`nucleus-frontend`	The nucleus-frontend pods host the frontend API server that routes all requests to the Domino application. Its logs will contain details about HTTP requests to Domino from the application or another API client. If you see errors in Domino with HTTP error codes like 500, 504, or 401, you can find corresponding logs here.
`nucleus-dispatcher`	The nucleus-dispatcher pod hosts the Domino scheduling and brokering service that sends user execution pods to Kubernetes for deployment. Errors in communication between Domino and Kubernetes result in corresponding logs from this service.
`keycloak`	The keycloak pod hosts the Domino authentication service. The logs for this service will contain a record of authentication events, including additional details on any errors.
`cluster-autoscaler`	This pod hosts the open-source Kubernetes cluster autoscaler, which controls and manages autoscaling resources. The logs for this service contain records of scaling events, both scaling up new nodes in response to demand and scaling down idle resources, including additional details on any errors.

nucleus-frontend

The nucleus-frontend pods host the frontend API server that routes all requests to the Domino application. Its logs will contain details about HTTP requests to Domino from the application or another API client. If you see errors in Domino with HTTP error codes like 500, 504, or 401, you can find corresponding logs here.

nucleus-dispatcher

The nucleus-dispatcher pod hosts the Domino scheduling and brokering service that sends user execution pods to Kubernetes for deployment. Errors in communication between Domino and Kubernetes result in corresponding logs from this service.

keycloak

The keycloak pod hosts the Domino authentication service. The logs for this service will contain a record of authentication events, including additional details on any errors.

cluster-autoscaler

This pod hosts the open-source Kubernetes cluster autoscaler, which controls and manages autoscaling resources. The logs for this service contain records of scaling events, both scaling up new nodes in response to demand and scaling down idle resources, including additional details on any errors.

Compute grid namespace

This namespace hosts user executions plus Domino environment builds. The installer configuration file defines the name of this namespace at namespaces.compute.name.

Logs in this namespace correspond to ephemeral pods hosting user work. Each pod contains a user-defined environment container, whose logs are execution logs. There are additional supporting containers in the pods, and their logs might contain additional information about errors or behavior seen in specific Domino executions.

Domino advises that you aggregate and keep at least 30 days of logs to facilitate debugging. You can harvest these with a variety of Kubernetes log aggregation utilities, including:
Loggly
Splunk
New Relic

Next steps

Project audit logs for project-level auditing including user permissions and Dataset-related actions.
Data Source audit logs.