Proactively prevent overspending on compute and storage by setting budgets and sending alerts to team members approaching their limits.
Budgets help you control costs for organizations and Projects by preventing overspending. By default, all Projects and organizations have no limit.
To set and configure your budgets:
-
Go to Admin > Advanced > Cost Budgets and Alerts.
-
In the Default Limits section, you can set a default monthly (by calendar month) budget that applies to all organizations or Projects.
-
Use Budget Overrides to change the default budget limit for a specific organization or Project.
Proactive alerts notify stakeholders when expenses for their organizations or Projects are nearing the predetermined budget threshold. These alert settings are consistent across all Projects and organizations and are delivered via both Domino notifications and email. By default, alerts are not configured.
To set and configure alerts:
-
Go to Alert Settings and select Alert on 75% and 100% of budget thresholds to enable alerts.
-
Go to Advanced > Email Settings to configure your email and allow email alerts.
If 75% of a budget is reached, Domino sends daily notifications.
|
Important
| With the FinOps trial license, FinOps only provides notifications for the first 3 months. Contact your Domino representative about upgrading to an enterprise license. |
To save money on storage expenses and monitor storage usage, admins can also set limits and quotas on the amount of storage used by Datasets.
To see all configuration keys related to read-write Datasets, see Central Configuration.
Set Dataset snapshot limits
Use the Central Configuration key com.cerebro.domino.dataset.quota.maxActiveSnapshotsPerDataset to limit the number of snapshots per Dataset.
When a Dataset reaches the snapshot limit, users receive an error message when they attempt to create an additional snapshot. You must delete old snapshots or increase the limit before writing additional snapshots.
Override snapshot limits
You can authorize individual Projects to ignore the snapshot limits set by com.cerebro.domino.dataset.quota.maxActiveSnapshotsPerDataset:
-
In the Project where you want to override limits, go to Settings > Hardware & Environment.
-
Select the Ignore Dataset Limits checkbox.
|
Important
| The Ignore Dataset Limits checkbox has no bearing on quotas covered in the next section. |
Dataset sizing
Large Datasets can be sized quickly and accurately without timing out. When a user interacts with a Dataset by uploading a file, deleting a file, creating a snapshot, or ending a run, the Dataset is queued to be sized in the background.
To avoid multiple expensive calls to size the Dataset, Domino waits 7 days between each sizing request. This interval can be updated by modifying the config key com.cerebro.domino.dataset.snapshotSizingPeriod. For more information, see Central Configuration.
Dataset quota limits
You can also set Dataset quotas to notify users if their Datasets are consuming too much storage. Quotas are "soft" quotas, which means that they never block users from creating more Datasets or snapshots.
Set quotas by navigating to Admin > Datasets.
There are two types of Dataset quotas:
-
Global quota- Creates a limit that applies to every user in the deployment. -
Quota override- Sets individual limits on specific users and overrides theglobal quota.
|
Note
| Quota overrides only work when a global quota is also configured. |
Dataset quota calculations
Domino calculates quotas based on a user’s total Dataset storage footprint comprised of all the Datasets they own, except for Datasets that are in the following states:
-
MarkedForDeletion -
DeletionInProcess -
Deleted -
Failed
A Dataset’s size is defined as the sum of the size of each snapshot in the Dataset, with the exception of deleted snapshots.
If a Dataset has multiple owners, then the size of that Dataset counts towards the quota of each owner. For more information, see Dataset Roles.
Quota notifications
As a user reaches certain thresholds of their quota, they receive notifications, emails, and UI warnings on Dataset pages. Thresholds can be configured and are listed in the Read-write Datasets configuration keys.
When a user reaches a threshold, actions are triggered by the following events:
-
Uploading files to a Dataset (both UI and CLI)
-
Creating a snapshot (both UI and CLI)
-
Creating a Dataset from a snapshot (both UI and CLI)
-
Deleting files from a Dataset in the UI
When a user receives an email about their Dataset storage quota, admins automatically receive a similar email. You can also add additional recipients under Quota Notifications.
See other ways Domino FinOps can help you optimize the performance of your AI projects while managing cloud expenditures:
-
Automatically allocate usage-based costs to Projects, organizations, and users.
-
Optimize Dataset storage usage to reduce cloud storage costs.
-
Generate chargeback/showback reports and use APIs to facilitate cost recovery across organizations and engage with high-expenditure users to optimize infrastructure usage.