Backups are bundled into tarballs containing the mongo, Git, keycloak, vault, experiment management (MLflow) tracking database, and model secrets, as well as config files for the backup.
The bundles use the filename convention YYMMDD-HHMMSS.tar.gz, and all content in the tarball is in a directory with the same timestamp format. If you created a backup on September 22nd, 2023 at 7:02:27 pm, the tarball
20230922-190227.tar.gz contains the following:
-
20230922-190227/
-
20230922-190227/config.local.yaml (Config file for local backup)
-
20230922-190227/config.remote.yaml (Config file for remote backup)
-
20230922-190227/config.transfer.yaml (Additional config file for remote backup when using the
--transferoption) -
20230922-190227/git.tar.gz (tarball of Git repos and critical user project metadata/version control)
-
20230922-190227/keycloak_scripts.tar.gz
-
20230922-190227/keycloak-postgres_archive_local-backup.gz (Postgres database dump)
-
20230922-190227/mlflow-postgres_archive_local-backup.gz (MLflow Postgres database dump)
-
20230922-190227/mongo_archive_local-backup.gz (Mongo archive dump of primary Domino application database)
-
20230922-190227/vault-k8s-secrets_local-backup.yaml (Kubernetes secrets and critical secrets to unlock Vault)
-
20230922-190227/vault-postgres_archive_local-backup.gz (Postgres database dump)
-
20230922-190227/model_secrets-k8s-secrets_local-backup.yaml (Kubernetes secrets and Environment variables for models)
-
|
Caution
|
The backup bundle only contains data from the Kubernetes cluster. This process does not back up data that is stored in object storage (for example, AWS S3 buckets, Azure containers, GCP buckets) on shared storage like NFS or CIFS/SMB or in an external Docker Registry. Typically, the infrastructure management function operates object storage externally. This is reliable and often large. The infrastructure admin is responsible for the backup/restore strategy for this data. |
The default backup location varies depending on the infrastructure and which Domino was installed:
-
On AWS, Azure, and GCP a dedicated backup storage bucket or container is configured during Domino installation.
-
For on-premises deployments, backups are delivered to the shared storage mount.
The backup system delivers its backup bundle tarballs (YYYY/MM/YYMMDD-HHMMSS.tar.gz ) to the backup storage backend that is in use for a deployment. For on-premises deployments, use a shell into the importer pod in /opt/shared/backups to access this location.