About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Configure budgets and alerts

You can use Domino FinOps to establish budgets and send alerts to team members nearing their spending limits on compute and storage resources.

To prevent overspending on computing and storage, Domino FinOps allows you to aggregate usage data by Cost Center or Organization using billing tags, enabling chargebacks across different projects. Administrators can also set limits and quotas on the amount of storage allocated to Datasets.

Configure billing tags

FinOps allows you to aggregate Cost Center or Organization usage by billing tags within Domino, enabling chargebacks across multiple projects:

  1. Go to Admin > Manage resources > Cost tracking > Billing tags.

  2. In the Configure Billing Tags section, you can set the Billing Tag feature to Disabled, Optional, or Required for new projects.

  3. For Billing Tags, enter one or more Cost Center or Organization names and click Add Billing Tags.

After Domino creates the Billing Tags, you can filter available projects and assign billing tags to them in the Manually apply Billing Tags section.

Configure budgets to prevent overspending

Budgets help you control costs for organizations and projects by preventing overspending. By default, all projects and organizations have no limit.

Set and configure your budgets:

  1. Go to Manage Cost.

  2. Go to Admin > Manage resources > Cost tracking > Budget & alerts.

  3. In the Default Limits section, set a default monthly (by calendar month) budget that applies to all organizations, projects, or billing tags.

  4. Use Budget Overrides to change the default budget limits for specific organizations, projects, or billing tags.

Configure alerts to notify stakeholders

Regular alerts notify stakeholders when expenses for their organizations are nearing the predetermined budget threshold. These alerts are delivered through both Domino notifications and email. Alerts are not configured by default. Billing tag notifications are only sent by email, and require email configurations to be set.

Set and configure alerts:

  1. Go to Admin > Manage resources > Cost tracking > Budget & alerts.

  2. Under Alert Settings, select Alert on 75% and 100% of budget thresholds to enable alerts.

  3. Add additional emails to receive notifications.

  4. Go to Platform settings > Email to configure your email and allow email alerts.

If 75% of a budget is reached, Domino sends daily notifications.

Important
 With the FinOps trial license, FinOps only provides notifications for the first 3 months. Contact your Domino representative about upgrading to an enterprise license.

Configure limits per dataset

Admins can also set limits and quotas on the amount of storage used by Datasets to save money on storage expenses and monitor usage.

To see all configuration keys related to read-write Datasets, see Configuration records.

Set Dataset snapshot limits

When a Dataset reaches the snapshot limit, users receive an error message when they attempt to create an additional snapshot.

Use the com.cerebro.domino.dataset.quota.maxActiveSnapshotsPerDataset Configuration records key to limit the number of snapshots per Dataset.

You must delete old snapshots or increase the limit before writing additional snapshots.

Override snapshot limits

You can authorize individual projects to ignore the snapshot limits that you set:

  1. Go to the Project that you want to set snapshot limits on.

  2. Go to Settings > Hardware & Environment.

  3. Select the Ignore Dataset Limits checkbox.

Selecting Ignore Dataset Limits does not impact any quotas for which hard limits have been set.

Dataset sizing

When a user interacts with a dataset by uploading a file, deleting a file, creating a snapshot, or ending a run, the dataset is queued to be sized in the background. This allows large datasets to be sized quickly and accurately without timing out.

To avoid multiple expensive calls to resize the dataset, Domino waits seven days between each sizing request.

For more information, see Configuration records.

Dataset quota limits

You can set dataset quotas to notify users if their datasets are using too much storage. Quotas are "soft" quotas, meaning they never prevent users from creating more datasets or snapshots.

There are two types of dataset quotas. To set quotas:

  1. Navigate to Admin > Datasets.

  2. Choose the type of quota to set:

    • Global quota - Creates a limit that applies to every user in the deployment.

    • Quota override - Sets individual limits on specific users and overrides the global quota.

Quota overrides only work when a global quota is also configured.

Dataset quota calculations

Domino calculates quotas based on a user’s total dataset storage footprint, which includes all the Datasets they own, except for datasets in specific states:

  • MarkedForDeletion

  • DeletionInProcess

  • Deleted

  • Failed

The size of a dataset is determined by adding up the size of each snapshot within the dataset, excluding any deleted snapshots.

If a dataset has multiple owners, the size of that dataset is counted towards each owner’s quota.

Dataset Roles has more detailed information about how the roles work together.

Quota notifications

As a user reaches certain thresholds of their quota, they receive notifications, emails, and UI warnings on Dataset pages.

Thresholds can be configured and are listed in the Read-write Datasets configuration keys.

When a user reaches a threshold, actions are triggered by the following events:

  • Uploading files to a Dataset (both UI and CLI).

  • Creating a snapshot (both UI and CLI).

  • Creating a Dataset from a snapshot (both UI and CLI).

  • Deleting files from a Dataset in the UI.

When a user receives an email about their Dataset storage quota, admins automatically receive a similar email. You can also add additional recipients under Quota Notifications.

Next steps

Here are some other ways Domino FinOps can help you optimize the performance of your AI projects while managing cloud expenditures: