This topic covers best practices for compute environment management. As a Domino administrator, you must curate the environments used by your organization. A proactive approach to environment management can prevent sprawl, avoid repetition in environment creation, and equip users with the tools they have to succeed in Domino.
As an administrator, your objective is to find a balance between giving users the freedom to be agile in development, while also maintaining enough control that you don’t end up with duplicate or unnecessary environments. administrator and users can create an arbitrary number of environments in Domino. You’ll want to manage their creation so that you don’t end up with dozens of global environments and hundreds of user environments, which can make it hard for users to know which environments to use, and hard for you as an admin to maintain.
Don’t let your Domino look like this:
Domino recommends admins follow these best practices:
-
Limit global environments.
In our experience, the benefits of focusing on a small number of global environments with broad applications, outweighs the benefit of creating many niche global environments. When a user requests an environment with new capabilities, consider whether you can add their requested features to an existing global environment instead of creating a new one.
-
Use clear, descriptive names for your environments.
A common cause of environment duplication is a user who cannot tell that an existing environment meets his or her needs. Clear, descriptive names on all environments will make the whole catalog comprehensible to new users or admins, and makes Domino easier to work with and maintain.
-
Add comments to the Dockerfile for each environment.
You can add comments to a Dockerfile like this:
# This is a comment that can provide a helpful description of the code to follow RUN echo 'This is an executed Dockerfile instruction' # Here's a block that installs Ruby RUN \ apt-get update && \ apt-get install -y rubyInvest in a well-commented Dockerfile. Each section must a have clear heading and comments to explain its purpose and implementation.
-
Share responsibility for environment management
If you have multiple teams or departments doing separate work in Domino, they must be responsible for maintaining their own team-specific environments. Find an advanced user in each team and make him or her a deputy for environment management. This person must be responsible for planning and understanding the environments his or her team needs, and must work with you on implementation. This reduces the workload of the admins, and ensures that environments are designed by someone with context on what users need.
-
Keep global images up-to-date and comprehensive
You must strive to have global images that cover the majority of users' needs. Users should only have to make minor additions to global environments when creating their own user environments, such as installing a specific version of a package. You don’t want a situation where users are re-installing Python or making other major changes, as this will result in a bloated and poorly performing environment.
-
Avoid time-consuming image pulls by caching global environments on the executor machine image
You must cache your global environments in your executor template machine image. This ensures that each new executor starts up with the base Docker image for any environment already cached. If users are setting up environments that have base images very different from what is cached on the machine image, it can lead to long pull times when launching executors. Contact Domino Support for help modifying your machine image.
-
Clean up old or poorly maintained environments
Create a culture of tidiness around environment creation and management. Enforce a standard of quality in naming and Dockerfile commenting, and be assertive about pruning unnecessary environments. See the following section of this document for a walkthrough.
Over time, it’s inevitable that the number of environments in your Domino will grow. It’s valuable to do an occasional review to weed out unused environments, update active ones, and consolidate where possible. Depending on the size of your organization and your use of Domino, this might be a yearly or quarterly task.
Review current environment usage
As an administrator, you can see all environments being used across your deployment in the Environments Overview. The table of environments on this page can be sorted by the # of Projects column to get a quick understanding of which environments are in common use. You can also enter global in the search to filter for global environments.
Click the name of an environment to see Dockerfile details, as well as the list of projects and models using that environment. The list will also include the date of the last run for each project and model. Keep an eye out for user environments that make duplicate changes to global base environments, as well as unused or poorly-maintained environments.
Plan changes to global environments
Based on what you learned by reviewing existing environments, you must plan an updated set of global environments that include the tools and features frequently added by users. In same cases, it might be as simple as adding a few packages to an existing global environment. You can also create a new global environment when necessary, but Domino recommends erring on the side of larger, more consolidated environments. Doing so will make it easier for your users to choose an environment, and it will be easier for you to manage and maintain the collection of environments in your deployment.
Before executing your plan and changing the available global environments, it’s best to inform your users of the impending changes and solicit their feedback. Explain the changes to existing environments, announce the creation of new ones, and provide recommendations for which environment to use for various types of projects.
Sunset deprecated and unused environments
As an administrator, you can archive any environment that is not currently in use (that is, not currently set as the Project Default environment for any projects). This will remove it from the Environments menu. Historical runs will still reference an archived environment, so archiving does not break reproducibility. Use archiving to encourage adoption of new, up-to-date, and consolidated environments. Environments can be unarchived if necessary. An alternative to archiving environments is to rename them to indicate that they are End-of-Life.