Proactively prevent overspending on compute and storage by setting budgets and sending alerts to team members approaching their limits.
Budgets help you control costs for organizations and Projects by preventing overspending. By default, all Projects and organizations have no limit.
To set and configure your budgets:
Go to Admin > Advanced > Cost Budgets and Alerts.
For the default budget that applies to all organizations, specify a value for Org ($k) under Default Limits (per month). The same pattern applies to Projects.
To change the default budget limit for a specific organization or Project, specify it in the Budget Overrides section.
Alerts proactively notify stakeholders that their organization or Project is approaching a predefined budget limit. Alert settings are shared across all Projects and organizations and appear as both a Domino notification and email. By default, no alerts are set.
To set and configure alerts:
Go to Alert Settings and select Alert on 75% and 100% of budget thresholds to enable alerts.
Go to Advanced > Email Settings to configure your email and allow email alerts.
If 75% of a budget is reached, Domino sends daily notifications.
|With a free Kubecost license, FinOps can only provide notifications for the first 15 days of the month. To extend notifications, contact your Domino representative about upgrading to an enterprise Kubecost license.
To save money on storage expenses and monitor storage usage, admins can also set limits and quotas on the amount of storage used by Datasets.
To see all configuration keys related to read-write Datasets, see Central Configuration.
Use the Central Configuration key
com.cerebro.domino.dataset.quota.maxActiveSnapshotsPerDataset to limit the number of snapshots per Dataset.
When a Dataset reaches the snapshot limit, users receive an error message when they attempt to create an additional snapshot. You must delete old snapshots or increase the limit before writing additional snapshots.
You can authorize individual Projects to ignore the snapshot limits set by
In the Project where you want to override limits, go to Settings > Hardware & Environment.
Select the Ignore Dataset Limits checkbox.
|The Ignore Dataset Limits checkbox has no bearing on quotas covered in the next section.
Large Datasets can be sized quickly and accurately without timing out. When a user interacts with a Dataset by uploading a file, deleting a file, creating a snapshot, or ending a run, the Dataset is queued to be sized in the background.
To avoid multiple expensive calls to size the Dataset, Domino waits 7 days between each sizing request. This interval can be updated by modifying the config key
com.cerebro.domino.dataset.snapshotSizingPeriod. For more information, see Central Configuration.
You can also set Dataset quotas to notify users if their Datasets are consuming too much storage. Quotas are "soft" quotas, which means that they never block users from creating more Datasets or snapshots.
Set quotas by navigating to Admin > Datasets.
There are two types of Dataset quotas:
Global quota- Creates a limit that applies to every user in the deployment.
Quota override- Sets individual limits on specific users and overrides the
|Quota overrides only work when a global quota is also configured.
Domino calculates quotas based on a user’s total Dataset storage footprint comprised of all the Datasets they own, except for Datasets that are in the following states:
A Dataset’s size is defined as the sum of the size of each snapshot in the Dataset, with the exception of deleted snapshots.
If a Dataset has multiple owners, then the size of that Dataset counts towards the quota of each owner. For more information, see Dataset Roles.
As a user reaches certain thresholds of their quota, they receive notifications, emails, and UI warnings on Dataset pages. Thresholds can be configured and are listed in the Read-write Datasets configuration keys.
When a user reaches a threshold, actions are triggered by the following events:
Uploading files to a Dataset (both UI and CLI)
Creating a snapshot (both UI and CLI)
Creating a Dataset from a snapshot (both UI and CLI)
Deleting files from a Dataset in the UI
When a user receives an email about their Dataset storage quota, admins automatically receive a similar email. You can also add additional recipients under Quota Notifications.
See other ways Domino FinOps can help you optimize the performance of your AI projects while managing cloud expenditures:
Automatically allocate usage-based costs to Projects, organizations, and users.
Optimize Dataset storage usage to reduce cloud storage costs.
Generate chargeback/showback reports and use APIs to facilitate cost recovery across organizations and engage with high-expenditure users to optimize infrastructure usage.