For Domino Nexus deployments, remote data plane monitoring provides the capability to have metrics collected from Domino application components, the Kubernetes cluster, and host infrastructure running in the remote data plane, just as they are collected from the local cluster. This provides a similar level of visibility for the Domino administrator into remote workload resource usage than if the workload was running locally, and helps to reduce the time it takes to resolve remote workload issues.
In addition to the Prometheus server running in a non-Nexus Domino cluster, remote data plane monitoring will deploy and run a pod in each data plane at install time running OpenTelemetry Collector, also called Otel Collector.
In a remote data plane, the OpenTelemetry Collector pod is configured to scrape metrics from Domino components, Kubernetes, and the underlying infrastructure and then forward the collected metrics to the OpenTelemetry Collector running in the control plane. Metrics sent from each remote data plane have a data_plane_id label containing the ID of the data plane they were collected from.
The OpenTelemetry Collector running in the control plane (or local data plane) is responsible for receiving connections from OpenTelemetry Collectors running in each of the remote data planes and forwarding it to the Prometheus server. Once ingested into Prometheus, the remote data plane metrics are available for use in alerting and monitoring just like any other metric collected from the local data plane.
Communication between OpenTelemetry Collectors is secured via TLS and basic Auth authentication. The secret used for the authentication is stored in the secret named domino-otel-collector-http-auth.