Connectivity to data sources such as Snowflake is handled by the Domino data source proxy pods in the domino-platform namespace.
<user-id>$ prod-field % kubectl get pods -n domino-platform | grep -i proxy
datasource-proxy-5c4496685d-2dpkg 1/1 Running 0 21d
datasource-proxy-5c4496685d-mrjsk 1/1 Running 0 21d
Logs from those pods can help to troubleshoot connectivity to data sources.
<user-id>$ prod-field % kubectl logs datasource-proxy-5c4496685d-2dpkg -n domino-platform | grep -i snow
time="2023-06-16T12:18:30Z" level=error msg="Authentication FAILED" func="gosnowflake.(*defaultLogger).Errorln" file="log.go:242"
{"level":"error","ts":1686917910.709162,"caller":"server/auth.go:61","msg":"call error","error":"rpc error: code = InvalidArgument desc = 390913 (08004): Your free trial has ended and all of your virtual warehouses have been suspended. Add billing information in the Snowflake web UI to continue using the full set of Snowflake features.","stacktrace":"github.com/cerebrotech/datasource-proxy/internal/server.(*AuthnMiddleware).CallCompleted
/datasource-proxy/internal/server/auth.go:61
github.com/apache/arrow/go/arrow/flight.CreateServerMiddleware.func2
/go/pkg/mod/github.com/apache/arrow/go/arrow@v0.0.0-20211112161151-bc219186db40/flight/server.go:80
google.golang.org/grpc.(*Server).processStreamingRPC
/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1548
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1623
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921"}
<user-id>$ prod-field %
The following sections provide useful steps to troubleshoot: - Basic Domino health - Connectivity and latency - Workspaces and Jobs issues - Compute environment build issues - Model API issues - Distributed model monitoring issues