Domino data sources provide a mechanism to create and manage connection properties to a supported external data service. Data sources can be created by both Domino administrators and users, and can be shared among collaborators. Connection properties are stored securely and there is no need to install data source specific drivers or libraries. A tightly coupled library provides a consistent access pattern for both tabular and file based data.
To learn about using the Data API to work with data sources, see Data Source Use Cases.
One common configuration pattern is for Domino administrators to create, configure, and manage broadly-used data sources, which are then exposed to all or a subset of users.
-
Go to Admin > Data > Data Sources.
-
Click Create a Data Source.
-
Select your data store.
-
Enter the configuration details for accessing your data store.
The details vary depending on your data store type. Connect to External Data has instructions for each supported type.
-
Click Next.
-
Select the credential type:
-
Individual - each user is required to provide their own credentials before using a data source
-
Service Account - Domino administrators provide a set of credentials that will be automatically applied on behalf of users with permissions to a given data source. End users cannot access or extract the credentials.
NoteOAuth authentication is not supported for service accounts. OAuth-authenticated connections can be used for any execution type except Model APIs.
Data source credentials are stored securely in the Domino secret store which is backed by HashiCorp Vault.
-
-
Click Next.
-
Enter your credentials for data store authentication.
You can click Test Credentials to verify that authentication works, or click Skip for Now to continue.
-
Select whether Everyone can use this data source or just Specific users or organizations.
-
Click Finish Setup.
Alternatively, individual users can also create data sources directly when they need access to a more specific Data Source than what an admin may have seeded on the deployment.
-
Go to Data in the left nav or in your project.
Regardless of where the creation is initiated, the resulting data sources can be used in any project by users with the appropriate permissions.
-
Click Create a Data Source.
-
Select your data store.
-
Enter the configuration details for accessing your data store.
The details vary depending on your data store type. Connect to External Data has instructions for each supported type.
-
Click Next.
-
Select the credential type:
-
Click Next.
-
Enter your credentials for data store authentication.
You can click Test Credentials to verify that authentication works.
-
Select whether Only you can use this data source or whether it is shared with Specific users or organizations.
-
Click Finish Setup.
Data sources have global scope in a Domino deployment and are accessible to any user with the appropriate permissions in any project. Users can add data sources to a project explicitly (Add a data source on project Data page) or implicitly when a data source is used directly in code from a project. This allows users to have visibility into which data sources are used in each of their projects.
When multiple users collaborate on a project, it is possible that a data source used by one user is not properly configured for another. Domino will proactively surface such problems both from the project Data page as well as from the Data tab in Domino Workspaces.
Domino notifies users when they do not have permissions. When a user sees this message, they should request access from the data source owner.
When users have access but have not configured their individual credentials, they will also see a notification and will be able to add their credentials. For a given data source, individual credentials need to be added only once.
After a data source is properly configured, the Domino Data API allows users to retrieve data using a uniform interface without having to install drivers or data source specific libraries.
There are two types of data sources - Tabular and File-based with each exposing a slightly different mechanism for retrieving data. The simplest way to get started is with the automatically generated code snippet example.
For more detailed information, see the Domino Data API reference.
When using the Domino Data API from a Domino execution, user identity verification for the purposes of enforcing Domino permissions happens automatically. The library will first attempt to use a Domino JWT token, or, if not available, a user API key.
The following is a summary of the user identity that will be used for data source access based on Domino execution type.
-
Workspaces and Jobs - user who started the execution
-
Launchers - user who started the launcher regardless of who created the launcher
-
Domino Apps - user who published the app regardless of who is accessing the app
-
Model API - no user identity
For Model APIs and other advanced use cases that require establishing a different user identity it is possible to inject an API key into an execution through an environment variable, and then use it explicitly when retrieving a data source.
For more detailed information, see Custom Authentication from the API documentation.
Domino Training Sets allow you to persist dataframes for model training and other analysis. You can store and load multiple versions of a given dataframe from a training set, allowing you to connect a model to the specific version of a dataframe that was used to train it.
The dataframe used as the basis for a Training Set can be constructed using the result of a Domino Data Source query (as described above) or through any other construction method.
In addition to storing the underlying dataframe, training sets can be used to capture additional monitoring metadata. When this additional data is present, Training Set versions will be available as sources of baseline training data when publishing model APIs in Domino.
Training sets are only scoped to the projects in which they are created. Users with project contributor permissions can create, load, and delete Training Set versions in that project.
Training sets are available as an API-only feature. For information on how to use the API, see the Domino Data API documentation, especially these topics: