About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Google Kubernetes Engine (GKE) upgrade guide

This topic describes how to upgrade Kubernetes in your Google Kubernetes Engine (GKE) Domino deployment. GKE is hosted on Google Cloud Platform (GCP).

Important

Immediately after the Kubernetes upgrade, you must upgrade Domino to a compatible version of Kubernetes. Domino will not work as expected until this is completed.

For example, after upgrading Kubernetes to v1.22, you must upgrade to Domino v5.2 or later. Kubernetes v1.22 is not compatible with older versions of Domino. Similarly, after upgrading to Kubernetes 1.23 or 1.24, you must upgrade to Domino v5.3 or later.

Prerequisites

To upgrade Kubernetes in GKE, you must have a Unix or Linux terminal with the following:

Use environment variables to set the values of IDs, names, and labels. This simplifies the commands you’ll run while upgrading:

export CLUSTER_PROJECT=<The project name of the cluster>
export CLUSTER_NAME=<The name of the cluster>
export CLUSTER_REGION=<The region where the cluster is located>

Get cluster credentials

  1. If you aren’t already signed in with gcloud, run gcloud init.

  2. To protect your current configuration file from being overwritten, run export KUBECONFIG=$(pwd)/kubeconfig.

  3. Get the remote kubeconfig:

    gcloud container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_REGION --project $CLUSTER_PROJECT

Disable the disruption budgets

You must temporarily disable the PodDisruptionBudget (PDB) for certain services. To do this, back up the PDBs, delete the original PDBs, then update Kubernetes and restore the PDBs from the backup after the update is complete.

  1. Back up the PDBs:

    kubectl get pdb -n domino-platform -o yaml mongodb-arbiter > mongodb-arbiter.yml
kubectl get pdb -n domino-platform -o yaml mongodb-primary > mongodb-primary.yml
kubectl get pdb -n domino-platform -o yaml mongodb-secondary > mongodb-secondary.yml

  2. Delete the PDBs:

    kubectl delete pdb -n domino-platform mongodb-arbiter
kubectl delete pdb -n domino-platform mongodb-primary
kubectl delete pdb -n domino-platform mongodb-secondary

Determine the upgrade version

  1. Get the available GKE versions:

    gcloud container get-server-config \
  --region=${CLUSTER_REGION} \
  --flatten="channels" \
  --format="yaml(channels.channel,channels.validVersions)"

  2. Get the current master version:

gcloud container clusters describe ${CLUSTER_NAME} --region=${CLUSTER_REGION} --format="value(currentMasterVersion)"
Note
 You can only upgrade by one minor version at a time. For example, if you are on Kubernetes version 1.22, you can upgrade to version 1.23.

Update the cluster

  1. Upgrade the control plane. This might take several minutes to complete. Monitor the GCP console for the status of the cluster upgrade.

    gcloud container clusters upgrade $CLUSTER_NAME --project $CLUSTER_PROJECT --master --region $CLUSTER_REGION --cluster-version <version-number>

  2. Upgrade the node pools:

    gcloud container clusters upgrade $CLUSTER_NAME --project $CLUSTER_PROJECT --region $CLUSTER_REGION --node-pool=platform
gcloud container clusters upgrade $CLUSTER_NAME --project $CLUSTER_PROJECT --region $CLUSTER_REGION --node-pool=compute
gcloud container clusters upgrade $CLUSTER_NAME --project $CLUSTER_PROJECT --region $CLUSTER_REGION --node-pool=gpu

Restore the disruption budgets

Run these commands from the same folder as the backup:

kubectl apply -f mongodb-arbiter.yml
kubectl apply -f mongodb-primary.yml
kubectl apply -f mongodb-secondary.yml