Verify your deployment

This document describes how to verify a Cloud Deploy deployment.

You can configure Cloud Deploy and Skaffold to verify that an application that you have deployed to any target is working correctly. Verification is done using your own testing image, and you configure Cloud Deploy and Skaffold to run those tests after deployment finishes.

By default, deployment verification runs in the Cloud Deploy execution environment, but you can also configure it to run on the same cluster where the application is running.

How does deployment verification work?

1. You configure Skaffold for verification.

   This configuration identifies the container image or images to use to run tests, and the specific commands (script, for example) to run from that container image.

2. You configure one or more targets in your delivery pipeline for deployment verification.

   This configuration enables verification for workloads deployed to that target.

3. After a rollout is deployed (skaffold apply), Cloud Deploy runs the skaffold verify command in the Cloud Deploy execution environment.

   For deployments to Google Kubernetes Engine and GKE Enterprise, you can optionally run the verification container on the same cluster where the application container is running.

4. Skaffold invokes the test or tests specified in the verify stanza of your skaffold.yaml to run against the deployed application.

5. Success or failure of the tests executed indicates success or failure of the verification.

   • Verification success is determined by the executed container's exit code.

     0 indicates success. A non-zero exit code indicates failure. To generate the desired verification result, make sure the container exits with the appropriate exit code. If more than one container is executed as part of verification, they must all succeed for verification to succeed.

   • If the verification fails, the rollout also fails.

   • If a deployment fails during verification, you can see that by inspecting the rollout:

     

6. You can ignore or retry a failed verification.

   You can also terminate an in-progress verify job.

Components used for verification

The rollout resource includes the following objects, which support deployment verification:

• Phase

  The collection of operations (jobs) in a rollout that are logically grouped together, for example a deploy or a deploy and verify.

• Job

  The specific operation to be performed on a rollout, such as deploy or verify.

• Job run

  A child of the rollout resource, the job run is an instance of a job, for example an attempt to deploy.

For more about Cloud Deploy resources, see Cloud Deploy service architecture

Notifications generated by deployment verification

Cloud Deploy generates Pub/Sub messages and publishes them for the following events:

• Job run create, update, and delete

  These notifications are published to the clouddeploy-resources topic, and contain the following attributes:

  • Resource
  • ResourceType (JobRun)
  • Action (Create, Update, Delete)
  • ProjectNumber
  • Location
  • TargetId
  • DeliveryPipelineId
  • ReleaseId
  • RolloutId
  • JobRunId

The following is an example Pub/Sub message for a job run create, published to the clouddeploy-resources topic:

{
    "ackId": "UAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRAGAE8CKF15MFcrQVh9Dz4NGXJ9YXRiWRIJBkUHeF9cEQ1iXE5EB0nq0KDVV1dKXxYGAExQeVhbHQVoWVh0Bnn7h5nK-8HjYwk9OqKarPdtO4PY2fNHZiI9XhJLLD5-My5FQV5AEkw4G0RJUytDCypYEU4EISE-MD5FU0Q",
    "message": {
      "attributes": {
        "Action": "Create",
        "DeliveryPipelineId": "dv-pipeline",
        "JobRunId": "634f8c6f-30c3-49ca-af80-68dc24d4cc5d",
        "Location": "us-central1",
        "ProjectNumber": "253401481285",
        "ReleaseId": "test-release-100",
        "Resource": "projects/253401481285/locations/us-central1/deliveryPipelines/dv-pipeline/releases/test-release-100/rollouts/test-release-100-to-dev-0001/jobRuns/634f8c6f-30c3-49ca-af80-68dc24d4cc5d",
        "ResourceType": "JobRun",
        "RolloutId": "test-release-100-to-dev-0001"
      },
      "messageId": "5572937706805411",
      "publishTime": "2022-09-07T14:00:46.040Z"
    }
  },

• Job run start, succeed, and fail

  These notifications are published to the clouddeploy-operations topic, and contain the following attributes:

  • Resource
  • ResourceType (JobRun)
  • Action (Start, Succeed, Failure)
  • ProjectNumber
  • Location
  • TargetId
  • DeliveryPipelineId
  • ReleaseId
  • RolloutId
  • JobRunId
  • PhaseId
  • JobId
  • JobType (Deploy or Verify)

The following is an example Pub/Sub message for a failed job run, published to the clouddeploy-operations topic:

{
    "ackId": "RFAGFixdRkhRNxkIaFEOT14jPzUgKEUUBAgUBXx9cEFPdVhec2hRDRlyfWB9aVsbCAUXU3cJURsHaE5tdR-6xcvaS0NVb18UAgRFWndfXhMEblhfcy-fkK3HwvT9U0AvOemNgdZpe6jHiulvZiM9XxJLLD5-My5FQV5AEkw4G0RJUytDCypYEU4EISE-MD5FUw",
    "message": {
      "attributes": {
        "Action": "Failure",
        "DeliveryPipelineId": "dv-pipeline",
        "JobId": "verify",
        "JobRunId": "b389224a-c259-4a00-ab75-c22e48bc3136",
        "JobType": "Verify",
        "Location": "us-central1",
        "PhaseId": "stable",
        "ProjectNumber": "253401481285",
        "ReleaseId": "test-release-101",
        "Resource": "projects/253401481285/locations/us-central1/deliveryPipelines/dv-pipeline/releases/test-release-101/rollouts/test-release-101-to-dev-0001/jobRuns/b389224a-c259-4a00-ab75-c22e48bc3136",
        "ResourceType": "JobRun",
        "RolloutId": "test-release-101-to-dev-0001",
        "TargetId": "dev"
      },
      "messageId": "5573609905896436",
      "publishTime": "2022-09-07T15:35:37.906Z"
    }
  },

Configure Cloud Deploy for deployment verification

Enabling deployment verification for a Cloud Deploy target consists of adding a verify: true property to a given target (or targets) in a delivery pipeline progression, as shown in this example:

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
 name: my-demo-app
description: main application pipeline
serialPipeline:
 stages:
 - targetId: dev
   profiles: []
   strategy:
     standard:
       verify: true
 - targetId: prod
   profiles: []
   strategy:
     standard:
       verify: false

In this configuration, deployment verification is enabled on the dev target, but not on the prod target. verify: false is equivalent to omitting the verify property or the entire strategy stanza.

The verify operation is executed within its own execution environment. This execution environment can be configured for VERIFY the same as it can be for RENDER and DEPLOY.

Configure Skaffold for deployment verification

Enabling deployment verification for a target requires a verify stanza in the skaffold.yaml configuration file for your deployment. This configuration can be for a specific Skaffold profile, if you're using separate profiles per target.

Note: Your skaffold.yaml must identify an apiVersion of skaffold/v3alpha1 or greater.

This verify stanza identifies a container to run to do the verification— for example, an integration test.

The following is an example skaffold.yaml that includes a verify stanza:

apiVersion: skaffold/v4beta7
kind: Config
build:
  artifacts:
    - image: integration-test
      context: integration-test
manifests:
  rawYaml:
  - kubernetes.yaml
deploy:
  kubectl: {}
verify:
- name: verify-integration-test
  container:
    name: integration-test
    image: integration-test
    command: ["./test-systems.sh"]
- name: verify-endpoint-test
  container:
    name: alpine
    image: alpine
    command: ["/bin/sh"]
    args: ["-c", "wget #ENDPOINT_URL"]

This simple example shows a verify stanza identifying a container to use and a test script to run in that container. #ENDPOINT_URL in this example is just a placeholder for your applications URL, and is not an available Cloud Deploy environment variable.

Note: Using Skaffold to build the container for verification is not required. It's included here as an example only.

Run the verification container on the application cluster

By default, deployment verification runs in the Cloud Deploy execution environment. You can also configure Skaffold to run the verification container on the same cluster where your application is running. When you configure in-cluster verification in skaffold.yaml and enable verification on a target, the verification runs automatically in that target's cluster.

This ability is available for deployments to GKE and GKE Enterprise only, not for Cloud Run. Deployments to Cloud Run can only run verification in the Cloud Deploy execution environment.

In-cluster verification requires Skaffold version 2.3 or higher.

Note: For parallel deployments, verification is configured on the multi-target, and the verification container runs on each child target.

To run your verification container on the cluster, include an executionMode.kubernetesCluster stanza in your skaffold.yaml configuration file, inside the verify stanza for the specific verification container:

verify:
- name: 
  container:
    name: 
    image: 
    command: 
    args: 
  executionMode:
    kubernetesCluster:

Note: you can also include the jobManifestPath and overrides properties, under kubernetesCluster, to point to a manifest for your verify container, and any values you want to override. (overrides takes Kubernetes inline JSON with the values you want to replace.) Learn more.

The following is an example verify stanza that includes executionMode to invoke the verification container on the application cluster:

verify:
- name: integration-test-container
  container:
    name: integration-test-container
    image: integration-test-container
  executionMode:
    kubernetesCluster: {}

The executionMode stanza is optional, and if you omit it, Skaffold runs the verification container in the Cloud Deploy execution environment.

Retry the verification

When a verify job fails, you can retry the verification, creating a new job run:

gcloud deploy rollouts retry-job ROLLOUT_NAME \
             --job-id=JOB_ID \
             --phase-id=PHASE_ID \
             --delivery-pipeline=PIPELINE_NAME \
             --release=RELEASE_NAME \
             --region=REGION

Retrying the verification changes the state of the rollout from FAILED to IN_PROGRESS.

You can only retry a verification for a rollout whose verify job failed.

Available environment variables

Cloud Deploy provides and populates the following environment variables in the VERIFY execution environment, which you can use for your tests:

• ANTHOS_MEMBERSHIP

  For targets of type ANTHOS, the fully specified resource name of the Anthos membership.

• CLOUD_RUN_LOCATION

  For targets of type RUN, the region the Cloud Run service is deployed in.

• CLOUD_RUN_PROJECT

  For targets of type RUN, the project in which the Cloud Run service was created.

• CLOUD_RUN_SERVICE

  For targets of type RUN, the fully-specified resource name of the deployed Cloud Run service, for example projects/p/locations/us-central1/services/dev.

• CLOUD_RUN_SERVICE_URLS

  For targets of type RUN, the URL or URLs (comma-separated list) that end users will use to access your service. You can find these in the Cloud Run service details for your service, in the Google Cloud console.

• CLOUD_RUN_REVISION

  For targets of type RUN, the specific revision of the Cloud Run service.

• GKE_CLUSTER

  For targets of type GKE, the fully specified resource name of the Google Kubernetes Engine cluster, for example projects/p/locations/us-central1/clusters/dev.

• TARGET_TYPE

  The specific runtime type of the target. Either GKE, ANTHOS, or RUN.

• CLOUD_DEPLOY_LOCATION

  The region in which the execution environment is running.

• CLOUD_DEPLOY_DELIVERY_PIPELINE

  The ID delivery pipeline the execution environment is running.

• CLOUD_DEPLOY_TARGET

  The ID of the target the execution environment is running.

• CLOUD_DEPLOY_PROJECT

  The project number of the Google Cloud project in which the execution environment is running.

• CLOUD_DEPLOY_RELEASE

  The ID of the release in which the verify will run.

• CLOUD_DEPLOY_ROLLOUT

  The ID of the rollout that contains the jobs for verify.

• CLOUD_DEPLOY_JOB_RUN

  The ID of the job run that represents the current execution of the job.

• CLOUD_DEPLOY_PHASE

  The phase in the rollout that contains the job for verify.

Deploy parameters as environment variables

In addition to the environment variables listed in this section, Cloud Deploy can pass to your custom containers any deploy parameters you've set.

Learn more.