Kubernetes pod security

Domino leverages Kubernetes security contexts to deploy pods with the least privileges necessary for them to function.

By default, pods created by Domino run with the following settings:

Run as a non-root user with runAsRoot: false, runAsNonRoot: true, and privileged: false set.
Acquisition of additional system-level capabilities must be prevented with drop: ["ALL"].

Domino does not support a policy where readOnlyRootFilesystem is set.

In addition, due to inconsistency between SELinux support and volume relabeling, all Domino pods run with an SELinux type option spc_t. This setting purposefully bypasses volume relabeling due to the performance impact of volume relabeling while creating Domino user workloads.

Pods that do not adhere to the policies above, such as requiring root privileges or additional capabilities, are documented below with privileged indicating pods that need additional user or root privileges and capabilities indicated pods that may require additional system-level capabilities.

Pod Name Exceptions Description

Pod Name	Exceptions	Description
`aws-ebs-csi-driver-node`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order to provision and manage AWS EBS volumes for Domino. Will not be installed if AWS EBS is not configured as a storage provider during installation.
`aws-efs-csi-driver-controller`	`capabilities` `runAsRoot`	Requires access to host node resources in order manage AWS EFS volumes for Domino. Will not be installed if AWS EFS is not configured as a storage provider during installation.
`aws-efs-csi-driver-node`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order manage AWS EFS volumes for Domino. Will not be installed if AWS EFS is not configured as a storage provider during installation.
`csi-driver-smb-controller`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order to manage SMB volumes for Domino. Will not be installed if SMB volume management is not configured during installation.
`csi-driver-smb-linux`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order to manage SMB volumes for Domino. Will not be installed if SMB volume management is not configured during installation.
`dcgm-exporter`	`capabilities` `runAsRoot`	Requires access to host node resources in order to communicate with NVIDIA host engine and expose GPU metrics. Will only run on nodes with NVIDIA GPUs. An option to disable this service is available by setting gpu_monitoring_enabled to false in deploy.yaml
`docker-registry-cert-mgr`	`runAsRoot`	Requires access to host node resources in order to manage trusted certificates for Docker when Domino is configured to use a self-provided image registry. Will not be installed if an external Docker registry is configured during installation.
`domino-workbench-backup`	`capabilities` `runAsRoot`	Requires privileged access to filesystem resources for data backup.
`domino-workbench-backup-check`	`capabilities` `runAsRoot`	Requires privileged access to filesystem resources for data backup.
`domino-data-importer`	`capabilities` `runAsRoot`	Requires privileged access to filesystem resources for disaster recovery and data access. This service does not run by default and must be enabled by admins when needed.
`elasticsearch-tls`	`capabilities` `privileged` `runAsRoot`	Requires privileged access to system in order to set a mandatory system limit. If the system setting can be managed outside of Domino, elevated privileges can be removed by setting the following: `elasticsearch: enable_sysctl: false`
`falco`	`privileged` `runAsRoot`	Requires access to host node resources in order to audit system events using eBPF.
`fluentd`	`capabilities` `runAsRoot`	Requires access to host node resources in order to follow container logs which are used to view workload output within Domino.
`hephaestus-buildkit`	`capabilities` `privileged` `runAsRoot`	BuildKit powers container image building for the environments feature of Domino. By default, requires privileged access in order to utilize `overlay` mounts for performance during container image building. If your operating system supports rootless overlayfs natively, the privileges can be removed by setting the following: `image_building: rootless: true`
`image-caching-agent`	`runAsRoot`	Requires access to host node resources in order to communicate with the container runtime when caching container images for faster workload scheduling. Can be disabled by setting the following: `image_caching: enabled: false`
`istio-cni-node`	`privileged` `runAsRoot`	Requires access to host node resources in order to facilitate network setup for Istio. Will not be installed if Istio is not enabled or the CNI option is disabled.
`newrelic-infrastructure`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order to effectively monitor CPU, memory, network, and disk usage and report to New Relic. Will not be installed if New Relic is not configured for Domino during installation.
`newrelic-logging`	`capabilities` `runAsRoot`	Requires access to host node resources in order to follow container logs and report to New Relic. Will not be installed if New Relic is not configured for Domino during installation.
`nfs-client-provisioner`	`capabilities` `runAsRoot`	NFS Client Provisioner requires privileged access to filesystem resources in order to manage NFS volumes for Domino. Will not be installed if NFS is not configured as a storage provider during installation.
`nvidia-device-plugin`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order to communicate with the Kubernetes device API and expose NVIDIA GPU resources to the cluster. Will not be installed if GPU capabilities are not configuring during installation.
`nvidia-device-plugin-mps-control-daemon`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order to communicate with the Kubernetes device API and expose NVIDIA GPU resources to the cluster. Will not be installed if GPU capabilities are not configuring during installation.
`nvidia-driver-installer`	`capabilities` `privileged` `runAsRoot`	Requires access to host node resources in order to install and manage NVIDIA drivers for GPU-enabled instance types. Only required for installations on GKE clusters.
`openebs`	`capabilities` `runAsRoot`	Requires access to host node resources in order to manage storage volumes. Will not be installed if OpenEBS is not configured as a storage provider during installation.
`smarter-device-manager`	`runAsRoot`	Requires access to host node resources in order to communicate with the Kubernetes device API and expose FUSE devices to the cluster.

aws-ebs-csi-driver-node

capabilities
privileged
runAsRoot

Requires access to host node resources in order to provision and manage AWS EBS volumes for Domino. Will not be installed if AWS EBS is not configured as a storage provider during installation.

aws-efs-csi-driver-controller

capabilities
runAsRoot

Requires access to host node resources in order manage AWS EFS volumes for Domino. Will not be installed if AWS EFS is not configured as a storage provider during installation.

aws-efs-csi-driver-node

capabilities
privileged
runAsRoot

Requires access to host node resources in order manage AWS EFS volumes for Domino. Will not be installed if AWS EFS is not configured as a storage provider during installation.

csi-driver-smb-controller

capabilities
privileged
runAsRoot

Requires access to host node resources in order to manage SMB volumes for Domino. Will not be installed if SMB volume management is not configured during installation.

csi-driver-smb-linux

capabilities
privileged
runAsRoot

Requires access to host node resources in order to manage SMB volumes for Domino. Will not be installed if SMB volume management is not configured during installation.

dcgm-exporter

capabilities
runAsRoot

Requires access to host node resources in order to communicate with NVIDIA host engine and expose GPU metrics. Will only run on nodes with NVIDIA GPUs. An option to disable this service is available by setting gpu_monitoring_enabled to false in deploy.yaml

docker-registry-cert-mgr

runAsRoot

Requires access to host node resources in order to manage trusted certificates for Docker when Domino is configured to use a self-provided image registry. Will not be installed if an external Docker registry is configured during installation.

domino-workbench-backup

capabilities
runAsRoot

Requires privileged access to filesystem resources for data backup.

domino-workbench-backup-check

capabilities
runAsRoot

Requires privileged access to filesystem resources for data backup.

domino-data-importer

capabilities
runAsRoot

Requires privileged access to filesystem resources for disaster recovery and data access. This service does not run by default and must be enabled by admins when needed.

elasticsearch-tls

capabilities
privileged
runAsRoot

Requires privileged access to system in order to set a mandatory system limit. If the system setting can be managed outside of Domino, elevated privileges can be removed by setting the following:

elasticsearch:
  enable_sysctl: false

falco

privileged
runAsRoot

Requires access to host node resources in order to audit system events using eBPF.

fluentd

capabilities
runAsRoot

Requires access to host node resources in order to follow container logs which are used to view workload output within Domino.

hephaestus-buildkit

capabilities
privileged
runAsRoot

BuildKit powers container image building for the environments feature of Domino. By default, requires privileged access in order to utilize overlay mounts for performance during container image building. If your operating system supports rootless overlayfs natively, the privileges can be removed by setting the following:

image_building:
  rootless: true

image-caching-agent

runAsRoot

Requires access to host node resources in order to communicate with the container runtime when caching container images for faster workload scheduling. Can be disabled by setting the following:

image_caching:
  enabled: false

istio-cni-node

privileged
runAsRoot

Requires access to host node resources in order to facilitate network setup for Istio. Will not be installed if Istio is not enabled or the CNI option is disabled.

newrelic-infrastructure

capabilities
privileged
runAsRoot

Requires access to host node resources in order to effectively monitor CPU, memory, network, and disk usage and report to New Relic. Will not be installed if New Relic is not configured for Domino during installation.

newrelic-logging

capabilities
runAsRoot

Requires access to host node resources in order to follow container logs and report to New Relic. Will not be installed if New Relic is not configured for Domino during installation.

nfs-client-provisioner

capabilities
runAsRoot

NFS Client Provisioner requires privileged access to filesystem resources in order to manage NFS volumes for Domino. Will not be installed if NFS is not configured as a storage provider during installation.

nvidia-device-plugin

capabilities
privileged
runAsRoot

Requires access to host node resources in order to communicate with the Kubernetes device API and expose NVIDIA GPU resources to the cluster. Will not be installed if GPU capabilities are not configuring during installation.

nvidia-device-plugin-mps-control-daemon

capabilities
privileged
runAsRoot

nvidia-driver-installer

capabilities
privileged
runAsRoot

Requires access to host node resources in order to install and manage NVIDIA drivers for GPU-enabled instance types. Only required for installations on GKE clusters.

openebs

capabilities
runAsRoot

Requires access to host node resources in order to manage storage volumes. Will not be installed if OpenEBS is not configured as a storage provider during installation.

smarter-device-manager

runAsRoot

Requires access to host node resources in order to communicate with the Kubernetes device API and expose FUSE devices to the cluster.

All user-generated pods from workspaces, jobs, apps, or on-demand compute clusters run with a user and capabilities according to the Central Configuration entry for com.cerebro.domino.computegrid.kubernetes.nonRootExecutions.enabled. See Central Configuration and non-root executions release notes for details.

All user-generated pods from models run as a non-root user but with privilege-escalation capability.

Please see Pod Security Policies for an overview of how policies are enforced and specific information about the removal of Pod Security Policies in Kubernetes v1.25.

Pod Security Policies

A Pod Security Policy (PSP) is a cluster-level resource that controls security-sensitive aspects of the pod specification. PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields.

Pod security policies serve two purposes in helping secure a Domino installation:

They prevent the admission (creation) of pods in the cluster if their security stance does not meet the defined policy.
They mutate (modify) pods that have been allowed by the policy to set defaults matching the PSP (where no values are set).

Pod Security Policy deprecation and removal

As of Kubernetes 1.21, PSPs are a deprecated feature. In Kubernetes 1.25, support for PSPs has been completely removed.

In order to support the removal of Pod Security Policies, Domino 5.4 and above do not create per-service PSPs as part of the installation. Instead, all pods define an explicit security context. This removes the need for duplicate modification of security contexts during pod creation by PSPs.

Domino will not support a replacement for runtime policy enforcement. Because Domino controls the definition of all pods created in its namespaces, PSP enforcement of security boundaries after installation (during “runtime”) is not necessary. Domino can control and report (during “build time”) the security context of every pod to ensure all services meet our security standards. See Kubernetes Pod Security for an overview with specifics and exemptions.

OpenShift

OpenShift maintains its own version of pod security policies named "Security Context Constraints" and Red Hat has laid out a different evolution for that feature going forward from upstream Kubernetes.

Domino does not deploy any custom security contexts but makes use of OpenShift-provided SCCs (anyuid and privileged) bound to all Domino namespaces. Any additional SCCs will cause the installation of Domino to fail.

In addition, UID remapping is not supported in Domino and all services run as the user ID defined in their container images.

Kubernetes < 1.25 support

Domino binds a blanket unrestricted policy for clusters where PSPs are still enabled. Many cluster providers, such as EKS, already do this. If Domino is the only application deployed in a cluster running Kubernetes < 1.25, it is advisable to turn off pod security policy support entirely.