Git-based Projects with CodeSync

Note

We recommend using Git-based projects in Domino to take advantage of our CodeSync technology if:

  • you have at least a beginner’s proficiency with Git,

  • you have experience using hosted version control (GitHub, Bitbucket, etc.), and

  • you frequently collaborate with multiple data scientists on projects.

To learn more about common workflows using Git and branching, we recommend the following resources: Git Feature Branch Workflow.

If you’re not familiar with Git or hosted version control, or if you’re primarily looking to reproduce data science research, then a project that uses the Domino File System (DFS) may be a better fit for you.

Git-based projects with CodeSync are a beta feature. If you’d like to provide feedback on Git-based projects, please file a ticket.

Overview

Git-based projects provide a full CodeSync experience for your code by using Git and a Git service provider of your choice. Integrated CodeSync technology ensures that all of the common Git workflows, like committing, pushing changes, and more, are available to you natively within workspaces launched in Git-based projects. This makes it easy for you to engage in version controlled, code-based collaboration with fellow project team members, all from within Domino. Git-based projects also organize your projects’ assets as either Code, Data, or Artifacts, an organizational structure intended to support common data science workflows.




Create a Git-based project

If you plan on using a private Git repository to store your code, then you’ll first need to add the corresponding Git credentials in your Domino account settings prior to creating your project. After adding credentials, you’ll be able to easily create a Git-based project in Domino, thereby enabling CodeSync experience.

If you plan on using a public Git repository to store your code, then you won’t need to add any Git credentials.

To create a Git-based project:

  1. Click on Projects in the Domino sidebar menu and then click the New Project button. A project creation modal will appear.

gbp-2

  1. Enter a name for your project.

  2. Set your project’s visibility.

  3. Advance to the wizard’s next screen by clicking the “Next” button.

gbp-13

  1. Under “Hosted By”, select “Git Service Provider (beta)”. On selection, further fields will appear beneath the “Hosted By” field.

  2. Under “Git Service Provider”, select the provider currently hosting the repository you wish to import (the “target repository”).

  3. Under “Git Credentials”, select credentials authorized to access the target repository.

  4. Under “Repository”, input the url for the target repository. If you are using a PAT credential with Github or GitLab, the option to create your own repository will also appear.

  5. Click Create

gbp-14

Attention

If the repository you’re using to store your code contains one or more files exceeding 2 GB in size, Domino will create your Git-based project, but you might fail at workspace setup. Consider using Domino File System in that case.

You can use the following tool to check the total size of a Git repository, as well as the size of individual files within the repository: git sizer.

Attention

During the project creation process, one can create a completely new repository. To do so, select the “Create a new repo” option in the Repository dropdown field. Then specify the Owner/Organization associated with the repository and its visibility.




Code, Data, & Artifacts

In Domino, the Domino File System (DFS) is the traditional way of storing a project’s assets. DFS-based projects organize all of your project’s assets as either Data or Files. Git-based projects, however, organize your project’s assets as either Code, Data, or Artifacts, and apply CodeSync experience to Code assets.

Code – This section of your Git-based project organizes and lists all of the Git-based repositories used to store your project’s code, as well as any additional imported repositories. For more information, see Git repositories in Domino. Files within any of these repositories can be accessed from within a Domino workspace via CodeSync technology.

The common Git workflows, like committing, pushing, pulling, and more, are available to you when interacting with your code from within a Domino workspace. For more information, see Using Git in your workspace.

Note

The default working directory for your code will be /mnt/code.

You can browse the directories of linked Git repositories natively from the Code section on project page.

Note

As of Domino 4.5.0 release, this ability is provided for the default branch of the Git repository (switching branches is not currently supported).

Note that accessing the file structure may take up to 30 seconds on initial load, depending on the size of the repository.

gbp-15

Data – Similar to DFS projects, this section of your Git-based project organizes and lists all data sources used in your project, including Domino datasets, external data volumes, and dataset scratch spaces. For more information on how to use data with your project, please refer to the Domino datasets documentation.

Artifacts – Git-based projects introduce “Artifacts”. Artifacts are typically results or products from your research and analysis, like plots, charts, serialized models, and more. You can organize these outputs in this section, as well as import artifacts from other projects.

gbp-1




Running jobs

Warning

If you run a Job in a Git-based project, CodeSync ensures that only artifacts will be automatically synced and saved to the Domino File System (DFS). Code, on the other hand, will not be automatically synced / pushed to the Git repository being used for the Git-based project. This is intentional and intended to support the “Code”, “Data”, and “Artifacts” workflow. To learn more, see running jobs.




Working with artifacts in your workspace

Attention

All files in Artifacts are saved exclusively to the Domino File System (DFS). If you do not want to save a particular asset to the Domino File System, we recommend that you do not save it as an artifact. To learn more, see Syncing your work to Domino.

Artifacts are results from your research, like plots, charts, serialized models, and more. In Domino, you can save these results in the Artifacts section of your project.

Saving artifacts and pushing changes

  1. Click on the File Changes option in the sidebar of your workspace.

  2. Under “Artifacts”, view changes by expanding “File Changes”.

  3. Enter a commit message.

  4. Click Sync to Domino. Domino will save your artifacts to the Domino File System (DFS).

gbp-8

Pulling changes

To pull the latest artifacts (from the Domino File System) into your workspace:

  1. Click the File Changes option in the sidebar menu of your workspace.

  2. Under the “Artifacts” section, click Pull. Domino will pull the latest changes into your workspace.

gbp-9