Configure node upgrade strategies

Standard

---

As a platform administrator, you can configure a node upgrade strategy to tune how GKE upgrades the nodes in your Google Kubernetes Engine (GKE) clusters. To learn more about node upgrade strategies, see Node upgrade strategies.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Configure a node upgrade strategy

When configuring your cluster's node pools, you can select and configure one of the supported node upgrade strategies, namely surge or blue-green. Using these upgrade strategies lets you optimize the node pool upgrade process based on your cluster environment's needs.

Note: By default, node pools use the surge upgrade strategy with maxSurge=1 and maxUnavailable=0.

Configure surge upgrades

Surge upgrades allow you to change the number of nodes GKE upgrades at one time and the amount of disruption an upgrade makes on your workloads.

The max-surge-upgrade and max-unavailable-upgrade flags are defined for each node pool. For more information on choosing the right parameters, go to Optimize your surge upgrade configuration.

Note: All new node pools are automatically configured to use surge upgrades maxSurge=1 maxUnavailable=0. To tune your configuration, see Optimize your surge upgrade configuration.

Warning: For nodes that use Spot VMs, surge upgrade values are ignored because there is no availability guarantee. During an upgrade, old nodes are drained directly without waiting for surge nodes that use Spot VMs to be ready.

You can change these settings when creating or updating a cluster or node pool.

The following variables are used in the commands mentioned below:

• CLUSTER_NAME: the name of the cluster for the node pool.
• COMPUTE_ZONE: the zone for the cluster.
• NODE_POOL_NAME: the name of the node pool.
• NUMBER_NODES: the number of nodes in the node pool in each of the cluster's zones.
• SURGE_NODES: the number of extra (surge) nodes to be created on each upgrade of the node pool.
• UNAVAILABLE_NODES: the number of nodes that can be unavailable at the same time on each upgrade of the node pool.

Creating a cluster with specific surge parameters

To create a cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container clusters create CLUSTER_NAME \
    --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Creating a cluster with surge upgrade disabled

To create a cluster without surge upgrades, set the value for the max-surge-upgrade flag to 0.

gcloud container clusters create CLUSTER_NAME \
    --max-surge-upgrade=0 --max-unavailable-upgrade=1

Creating a node pool with specific surge parameters

To create a node pool in an existing cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container node-pools create NODE_POOL_NAME \
    --num-nodes=NUMBER_NODES --cluster=CLUSTER_NAME \
    --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Change surge upgrade settings for an existing node pool

To update the upgrade settings of an existing node pool, use the max-surge-upgrade and max-unavailable-upgrade flags. If you set max-surge-upgrade to greater than 0, GKE creates surge nodes. If you set max-surge-upgrade to 0, GKE doesn't create surge nodes.

gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Checking if surge upgrades are enabled on a node pool

To see if surge upgrades are enabled on a node pool, use gcloud to describe the cluster's parameters:

gcloud container node-pools describe NODE_POOL_NAME \
    --cluster=CLUSTER_NAME

If surge upgrades are enabled on the node pool, the strategy listed is SURGE.

Note: While recreating nodes does not require additional Compute Engine resources, surge upgrading nodes does. Resource allocation is subjected to Compute Engine quota. Depending on your configuration, this quota can limit the number of parallel upgrades or even cause the upgrade to fail. If you don't have any additional quota or reservation for the nodes in your node pool, don't use maxSurge.

For more information about quota, go to Ensure resources for node upgrades.

Configure blue-green upgrades

With blue-green node pool upgrades, you can control:

• BATCH_NODE_COUNT or BATCH_PERCENT: the size of batches of nodes that GKE drains at a time, meaning that the Pods are removed from the nodes. Default is BATCH_NODE_COUNT=1. If either of these settings are set to 0, GKE skips this phase and proceeds to the Soak node pool phase.
• BATCH_SOAK_DURATION: the time between each batch of nodes being drained.
• NODE_POOL_SOAK_DURATION: the amount of soak time for you to validate your workload on the new node configuration.

To learn more about how the phases of blue-green upgrades work, see The phases of blue-green upgrades.

The following variables are used in the commands listed in the next sections:

• CLUSTER_NAME: the name of the cluster for the node pool.
• NODE_POOL_NAME: the name of the node pool.
• NUMBER_NODES: the number of nodes in the node pool in each of the cluster's zones.
• BATCH_NODE_COUNT: the number of blue nodes to drain in a batch during the blue pool drain phase. Default is 1. If it is set to 0, the blue pool drain phase will be skipped.
• BATCH_PERCENT: the percentage of blue nodes to drain in a batch during the blue pool drain phase, expressed as a decimal between 0 and 1, inclusive. GKE rounds down to the nearest node, to a minimum value of 1 node, if the percentage isn't a whole number of nodes. If it is set to 0 the blue pool drain phase will be skipped.
• BATCH_SOAK_DURATION: the duration in seconds to wait after each batch drain. Default is 0.
• NODE_POOL_SOAK_DURATION: the duration in seconds to wait after completing drain of all batches. Default is 3600 seconds.

Creating a node pool with blue-green upgrade strategy

Create a node pool with blue-green upgrade default parameters

To create a node pool in an existing cluster with the blue-green upgrade strategy with the default parameters, use the following command:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-blue-green-upgrade

Create a node pool with blue-green upgrade using absolute node count batch sizes

To create a node pool with custom blue-green upgrade settings, use the parameter flags with the node pool creation command.

This command creates a node pool with the following customized blue-green configuration, using an absolute node count for the batch drains:

• BATCH_NODE_COUNT = 2
• BATCH_SOAK_DURATION = 10s
• NODE_POOL_SOAK_DURATION = 600s

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-blue-green-upgrade \
    --standard-rollout-policy=batch-node-count=2,batch-soak-duration=10s \
    --node-pool-soak-duration=600s

Create a node pool with blue-green upgrade using percentage-based batch sizes

This command creates a node pool with the following customized blue-green configuration, using a percentage for the batch drains:

• BATCH_PERCENTAGE = 25% (of the node pool size)
• BATCH_SOAK_DURATION = 10s
• NODE_POOL_SOAK_DURATION = 1800s

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-blue-green-upgrade \
    --standard-rollout-policy=batch-percent=0.25,batch-soak-duration=10s \
    --node-pool-soak-duration=1800s

Updating an existing node pool blue-green upgrade strategy

Update a node pool with blue-green upgrade default parameters

To update an existing node pool to the blue-green upgrade strategy, use the following command:

gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-blue-green-upgrade

Update a node pool with blue-green upgrade using absolute node count batch sizes

To update an existing node pool to the blue-green upgrade strategy with custom settings, use the parameter flags with the node pool creation command.

This command updates a node pool to use the following customized blue-green configuration, using an absolute node count for the batch drains:

• BATCH_NODE_COUNT = 2
• BATCH_SOAK_DURATION = 10s
• NODE_POOL_SOAK_DURATION = 600s

gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-blue-green-upgrade \
    --standard-rollout-policy=batch-node-count=2,batch-soak-duration=10s \
    --node-pool-soak-duration=600s

Update a node pool with blue-green upgrade using percentage-based batch sizes

This command creates a node pool with the following customized blue-green configuration, using a percentage for the batch drains:

• BATCH_PERCENTAGE = 25% (of the node pool size)
• BATCH_SOAK_DURATION = 10s
• NODE_POOL_SOAK_DURATION = 1800s

gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-blue-green-upgrade \
    --standard-rollout-policy=batch-percent=0.25,batch-soak-duration=10s \
    --node-pool-soak-duration=1800s

Switching back to surge upgrades

You can change the behavior of blue-green upgrades with settings, and control the upgrade process with commands.

However, if you want to use surge upgrades instead, run the following command to switch back to surge upgrades:

gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-surge-upgrade

Inspect the upgrade settings of a node pool

To inspect the current upgrade settings of a node pool, you can use the following command to describe the node pool:

gcloud container node-pools describe NODE_POOL_NAME \
    --cluster=CLUSTER_NAME

The following snippet is an example output of the command. The strategy field indicates the upgrade strategy in use: SURGE indicates that the surge upgrade strategy is enabled, and BLUE_GREEN indicates that the blue-green upgrade strategy is enabled.

upgradeSettings:
  blueGreenSettings:
    nodePoolSoakDuration: 1800s
    standardRolloutPolicy:
      batchNodeCount: 1
      batchSoakDuration: 10s
  strategy: BLUE_GREEN

This command also shows you the current phase of an in-progress blue-green upgrade. Learn more about checking the upgrade settings of a node pool.

What's next

• Learn more about node upgrade strategies.
• Learn how to manually upgrade your node pools.