Use Domino’s AI Gateway to access multiple external Large Language Model (LLM) providers securely within your Workspaces and Runs. The AI Gateway serves as a bridge between Domino and external LLM providers such as OpenAI or AWS Bedrock. By routing requests through the AI Gateway, Domino ensures that all interactions with external LLMs are secure, monitored, and compliant with organizational policies. This guide outlines how to use the AI Gateway to connect to LLM services while adhering to security and auditability best practices.
The AI Gateway provides:
-
Security: Ensures that all data sent to and received from LLM providers is encrypted and secure.
-
Auditability: Keeps comprehensive logs of all LLM interactions, which is crucial for compliance and monitoring.
-
Ease of access: Provides a centralized point of access to multiple LLM providers, simplifying the user experience.
-
Control: Allows administrators to manage and restrict access to LLM providers based on user roles and project needs.
The AI Gateway is built on top of an MLflow Deployments Server for easy integration with existing MLflow projects.
Query an endpoint
To query an endpoint:
-
Click on the Copy Code icon.
-
Paste the code in your Workspace and adjust the query to fit your needs.
Alternatively, you can use the MLflow Deployment Client API to create your own query.
|
Note
|
When using the MLflow Deployment Client, Domino only supports the predict() API endpoint. To fetch an endpoint or list all endpoints, use Domino’s Public API instead.
|
AI Gateway endpoints are central to AI Gateway. Each endpoint acts as a proxy endpoint for the user, forwarding requests to a specific model defined by the endpoint. Endpoints are a managed way to securely connect to model providers.
To create an AI Gateway endpoint in Domino, you must use the aigateaway endpoint in the Domino REST API.
For example, the following curl command creates an AI Gateway endpoint using gpt-4, hosted by openai, that can be used by all users in the Domino deployment:
curl -d
'{
"endpointName":"completions",
"endpointType":"llm/v1/completions",
"endpointPermissions":{"isEveryoneAllowed":true,"userIds":[]},
"modelProvider":"openai",
"modelName":"gpt-4",
"modelConfig":{"openai_api_key":"<OpenAI_API_Key>"}
}' -H "X-Domino-Api-Key:$DOMINO_USER_API_KEY" -H "Content-Type: application/json" -X POST https://<deployment_url>/api/aigateway/v1/endpoints|
Important
|
The endpointName must be unique.
|
See MLflow’s Deployment Server documentation for more information on the list of supported LLM providers and provider-specific configuration parameters.
Once an endpoint is created, authorized users can query the endpoint in any Workspace or Run using the standard MLflow Deployment Client API. For more information, see the documentation to Use Gateway Model APIs.
Endpoint permission management
You can configure AI Gateway endpoints to be accessible to everyone or a specific set of users and/or organizations.
In the API, this can be accomplished through the endpointPermissions field when you create an AI Gateway or update a request. See the curl request above for an example.
Secure credential storage
When creating an endpoint, you will most likely need to pass a model-specific API key (such as OpenAI’s openai_api_key) or secret access key (such as AWS Bedrock’s aws_secret_access_key). When you create an endpoint, all of these keys are automatically stored securely in Domino’s central vault service and are never exposed to users when they interact with AI Gateway endpoints.
The secure credential store helps prevent API key leaks and provides a way to centrally manage API keys, rather than simply giving plain text keys to users.
Update an AI Gateway endpoint
To update an AI Gateway endpoint, you must use the aigateaway endpoint in the Domino REST API. For example, after creating the endpoint above, you can update the model with the following command:
curl -d
'{
"modelName":"gpt-3.5"
}' -H "X-Domino-Api-Key:<Domino_API_Key>" -H "Content-Type: application/json" -X PATCH https://<deployment_url>/api/aigateway/v1/endpoints/completionsTo update permissions you will need User IDs, which can be retrieved from the Endpoint API.
Domino logs all AI Gateway endpoint activity to Domino’s central audit system. To see AI Gateway endpoint activity, go to Admin > Advanced > MongoDB and run the following command:
db.audit_trail.find({ kind: "AccessGatewayEndpoint" })You can refine this query to filter records by fields such as the user or endpoint name:
db.audit_trail.find({ kind: "AccessGatewayEndpoint", "metadata.accessedByUsername": "johndoe" })
db.audit_trail.find({ kind: "AccessGatewayEndpoint", "metadata.endpointName": "openai_completions" })Learn how to create AI Gateway endpoints as a Domino admin and fine-tune Foundation Models.