Using Recommendations for Infrastructure as Code

---

Overview

Google Cloud Policy Intelligence helps enterprises understand and manage their policies to reduce their risk. By providing more visibility and automation, customers can increase security without increasing their workload.

Recommender enables you to retrieve recommendations for Google Cloud resources, helping you to improve cloud security, save costs, and more. For a list of supported recommendations refer to the Recommender documentation. This tutorial describes using sizing recommendations for VM instances and Identity and Access Management (IAM) recommendations. Recommender uses machine learning to provide administrators with recommendations for removing unnecessary access to Google Cloud resources and resizing Compute Engine instances for more efficient resource utilization.

Each recommendation includes a suggested action and its impact. After reviewing recommendations for the identified impacts as well as other considerations specific to your environment, you can select the recommendations that you want to apply. You can apply recommendations manually from the Google Cloud console or you can apply them programmatically by integrating them into your Infrastructure as Code (IaC) pipeline.

IaC enables you to automate the creation of your Google Cloud resources.You must keep your IaC repository up-to-date and route changes that are made to the Google Cloud organization through it. IaC strategies in organizations generally prove to be beneficial when they are implemented with rigor and serve as the single version of truth for your cloud infrastructure. Keeping your IaC repository up-to-date is critical to prevent drifts between the version of your infrastructure that your IaC repository reflects and what you have in the organization.

IAM recommendations

Amongst other leading practices, a common one is the security principle of least privilege and a careful consideration of how changes to your organization are rolled out and synchronized with your IaC repository.

Sizing recommendation for VMs

Sizing recommendations help you lower costs by providing suggestions for resizing the machine type of your instances to more efficiently use instance resources

This tutorial describes how to architect and build an automation pipeline to apply a Policy Intelligence recommendation programmatically. As part of this automation pipeline, you will learn how to keep your IaC repository up-to-date with the changes that you decide to make to your Google Cloud organization, based on the VM sizing and IAM policy bindings recommendation that Recommender makes available.

This tutorial uses Hashicorp Terraform as the IaC tool, however the architectural patterns and components used in the described automation pipeline can be leveraged even if you are using a different IaC management tool such as Deployment Manager. You will need to modify the open source codebase made available with this tutorial to suit your specific IaC implementation.

This guide is targeted at architects, product owners and developers who might be responsible for administration, security and infrastructure planning of their Google Cloud.

Automation pipeline architecture

The following diagram shows the components you use in this automation pipeline.

A scheduled Cloud Scheduler job runs the Recommender Parser service. The service calls the Recommender API to retrieve Recommender recommendations for the projects that you specify. It then parses these VM sizing and IAM recommendations to map them to the configuration you have in your Terraform manifests. The service updates your IaC manifests to reflect these recommendations. It generates a pull request with the changes so that you can review the updates. Once you review and merge the pull request, a Cloud Build job rolls out the changes to your infrastructure in your Google Cloud organization.

Several ancillary Google Cloud services are used in the pipeline for the purposes of tracking processed recommendations, generating notifications on build completion and storing Terraform state. You will learn more about these services over the course of this tutorial.

The following list describes the component purpose and access control requirements:

Platform Intelligence Recommenders

Purpose: Generate security and VM sizing recommendations

Access control: The Google Cloud service account must have the required IAM permissions to retrieve recommendations using the Recommender API.

Review Recommender roles and permissions to select the most appropriate role applicable to the service account that you use to run the recommender-parser service.

• Permissions for IAM recommender
• Permissions for Compute Engine Instance sizing

Cloud Scheduler

Purpose: Cloud Scheduler triggers the Recommender Parser service. Cloud Scheduler enables you to setup multiple jobs invoking as many instances of the parser service as you need. Each invocation needs to pass in the following inputs

• List of projects for which recommendations should be processed
• Recommendation Type
• IaC repository name

Note: You might want to set up multiple scheduler jobs if you have more than one IaC repository in your enterprise. Generally this is a common pattern in large enterprises.

Access control: Create or identify a Google Cloud service account to use for the calls from Cloud Scheduler to your Recommender Parser service.

Grant the service account the Cloud Scheduler Service Agent role so that it can run Cloud Scheduler jobs. Additionally, grant the service account the Cloud Run Invoker role since that account invokes a Cloud Run service

See the documentation about configuring authenticated access for scheduler jobs for details.

Cloud Run service

Purpose:: The recommender-parser service is where all the processing logic resides. It has multiple routes, each of which serve a specific purpose:

• Parsing recommendations for each recommendation type.
• Updating the status of the recommendations being processed

Access control: Use IAM to manage access to this service

Additionally, assign the service to a dedicated service account. This ensures that only the service is able to invoke other services such as Firestore.

Hashicorp Terraform

Purpose: Terraform 0.12 is the IaC tool.

A Cloud Build builder for Terraform is used to invoke Terraform commands and the Cloud Build service account is used for that purpose.

Cloud Build

Purpose: Google Cloud Build automates the deployment of infrastructure based on the changes made to the IaC manifests per policy intelligence recommendations.

Access control: The Cloud Build service account must have the right set of permissions to interact with resources in your test project.

See the documentation for configuring a Cloud Build service account.

GitHub

Purpose: The IaC repository uses GitHub for source control. The IaC repository in GitHub is integrated with Cloud Build. When commits are made to the master branch, a Cloud Build job is triggered to run a set of preconfigured tasks.

Access control: You will need to generate SSH keys to enable access to your IaC repository.

Additionally, you need to generate a Personal Access Token to push commits to GitHub.

Note: SSH keys and a Personal Access Token are required when interacting with private repositories hosted in GitHub. If your repository is public (not recommended) this step does not apply to you.

Firestore

Firestore is a fully managed, scalable NoSQL document database that is used in this architecture to persist information related to the Recommendation IDs that are parsed by the Recommender Parser service, along with the corresponding details pertinent to Git commits.

The details that are persisted in Firestore play an integral role in the feedback loop that is part of the end to end pipeline. After picking up a recommendation generated by the Recommender API and prior to processing the recommendation, the service marks the recommendation's state to CLAIMED. After the recommendation is successfully applied, the service queries the database to retrieve the recommendation IDs that have been successfully applied by the Cloud Build job and changes the recommendation state to SUCCEEDED. If the Cloud Build Job fails, the recommendation state is changed to FAILED.

Access control: Refer to Firestore roles for details. The recommender-parser service reads data from Firestore and needs the roles/datastore.user role to do so.

Pub/Sub

Purpose: Cloud Build publishes messages on a Pub/Sub topic when your build's state changes, such as when your build is created, when your build transitions to a working state, and when your build completes.

The Pub/Sub topic to which Cloud Build publishes messages is called cloud-builds, and it is automatically created for you when you enable the Cloud Build API in your project.

Access control: Push subscriptions can be configured to provide an authentication header to allow service to authorize the request. Refer to Using Push Subscriptions for more details.

Objectives

• Build an automation pipeline to
  • Proactively monitor platform Policy Intelligence recommendations
  • Parse recommendations and apply updates to an existing IaC repository
• Learn how you can use a suite of Google Cloud services, Hashicorp Terraform and GitHub to build this pipeline.
• Understand the assumptions and best practices you need to keep in mind to build this pipeline
• Test the pipeline

Costs

In this document, you use the following billable components of Google Cloud:

• Cloud Run
• Cloud Build
• Compute Engine
• Cloud Storage
• Firestore
• Pub/Sub
• Cloud Scheduler
• Recommender

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

This tutorial assumes that you have a GitHub account, and are familiar with Git, Node.js, Terraform and Docker.

Release Notes and Assumptions

There is a lot of variability in how IaC tools and manifests are used.

Review the following information to determine how this tutorial can fit your IaC pipeline and what kinds of changes might be required.

• This pipeline uses Terraform ver. 0.12. Significant changes in HCL configuration syntax or changes to the structure of the Terraform state file might introduce breaking issues.
• This pipeline assumes that the IaC directory structures are not nested and that one IaC repository manages resources in one or more Google Cloud projects.
• Terraform variables passed in as environment variables, command line arguments are not supported. The prototype assumes declarative configuration of Terraform variables in a tfvars file.
• Recommender generates IAM recommendations when a subset of permissions for a role have not been used for 60 days and VM sizing recommendations follow a similar pattern. For the purposes of this tutorial, sample recommendation payloads have been provided that can be used to test the pipeline.
• Loops within Terraform are not supported in this release
• Terraform Modules are not supported. The codebase is open source and it is assumed that you will make any necessary specific enhancements to the parsing flow to suit your directory structure and usage of modules.

The current version of the open source recommender parser service is aligned to the following known limitations of IAM recommendations:

• Recommendations can only be made for IAM policy bindings that are:
  • At the project level
  • Associated with user accounts and user-managed service accounts
• IAM recommendations support basic roles and predefined roles only. Custom roles and conditional bindings cannot be evaluated or recommended.
• Recommended roles only contain a subset of the current role's permissions. No new permissions are introduced by a recommended role.

Prerequisites

1. Select or create two Google Cloud projects.

   Go to the project selector page

   • A build project that hosts and runs the automation pipeline.
   • A test project that hosts Google Cloud resources used to test the automation pipeline.

2. Make sure that billing is enabled for your Google Cloud project.

3. In the test project, enable the Recommender and Compute Engine API.

   Enable the APIs

4. In the build project, enable the Cloud Run, Firestore, Pub/Sub and Cloud Scheduler, IAM and CloudResourceManager APIs.

   Enable the APIs

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. See Cleaning up for more detail.

Set up your environment

1. In Google Cloud console, select your build project.

2. In the Google Cloud console, go to Cloud Shell.

   Go to Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed, and with values already set for your current project. It can take a few seconds for the session to initialize.

   Use Cloud Shell for all of the terminal commands in this tutorial.

3. Create an environment variable to hold the project number for your build project using the command below:

   export BUILD_PROJECT_ID=$DEVSHELL_PROJECT_ID
   

4. Create an environment variable to hold the project number for your test project . Copy the test project ID manually and replace PROJECT-ID with it,

   export TEST_PROJECT_ID=PROJECT-ID

5. You assign default settings for values that are used throughout the tutorial, such as region and zone. In this tutorial, you use us-central1 as the default region and us-central1-b as the default zone.

6. Set the default region and zone for this tutorial by running the following command:.

   gcloud config set compute/zone us-central1-b --project $BUILD_PROJECT_ID
   gcloud config set compute/zone us-central1-b --project $TEST_PROJECT_ID
   

7. Set your build project as the default project:

   gcloud config set project $BUILD_PROJECT_ID
   

8. Create an environment variable calledBUILD_PROJECT_NUMBER for yourbuildproject number

   export BUILD_PROJECT_NUMBER=$(gcloud projects describe $DEVSHELL_PROJECT_ID --format='value(projectNumber)')
   

9. Clone the GitHub repository for this tutorial:

   Note: Execute all the commands in this tutorial in Cloud Shell unless specified otherwise.

Create Bucket for Terraform State

Create a Cloud Storage bucket in your build project to store the Terraform state file.

gcloud storage buckets create gs://recommender-tf-state-$BUILD_PROJECT_ID \
  --project=${BUILD_PROJECT_ID} --location=us-central1

Create a GitHub repository

Create a GitHub repository to serve as the sample IaC repository

1. Create a new private GitHub repository. This IAC-REPO-NAME repository serves as your IaC repository for the purposes of this tutorial

2. In the following setps, you will push the files in the sample-iac sub-directory of the cloned repository to your GitHub account.

   1. In Cloud Shell, copy the sample-iac directory to your home directory. You will use this directory to create a new local repository and push that to GitHub.

      cp -r recommender-iac-pipeline-nodejs-tutorial/sample-iac $HOME
      

   2. Navigate to the new directory

      cd $HOME/sample-iac
      

   3. Initialize the repository in your local machine.

      git init
      

   4. Add IAC-REPO-NAME as the remote repository, replace IAC-REPO-NAME and GITHUB-ACCOUNT with appropriate values

      git remote add origin https://github.com/GITHUB-ACCOUNT/IAC-REPO-NAME

   5. Replace the placeholders in the files in this repository with yourtestproject ID and the Terraform Cloud Storage bucket name.

      sed -i "s|__PROJECT_ID__|${TEST_PROJECT_ID}|g" ./terraform.tfvars
      
      sed -i "s|__STATE_BUCKET_NAME__|recommender-tf-state-$BUILD_PROJECT_ID|g" ./backend.tf
      

   6. Add, commit, and push to GitHub.

      git add .
      git commit -m "First Commit"
      git push origin master
      

   7. Sign in to your GitHub account when prompted.

Generate SSH keys for your repository

Set up SSH key authentication with your IaC repository in GitHub and upload the keys to Cloud Storage.

1. Generate SSH keys for your GitHub repository.

   1. Generate an SSH key pair. Replace [e-mail] with your GitHub email address. In Cloud Shell:

      ssh-keygen -t rsa -b 4096 -m PEM -C "[e-mail]"

   2. When you're prompted to "Enter a file in which to save the key," press Enter. This accepts the default file location.

   3. At the prompt to enter a passphrase, press Enter.

2. Make note of the directory SSH-KEYS-DIR that you save the downloaded SSH keys in. By default the location is$HOME/.ssh/

3. Copy the SSH public key that you generated to your GitHub repository as a Deploy Key.

   1. Copy the SSH public key that you generated in Cloud Shell. Replace SSH-KEYS-DIR with your directory path.

      cat SSH-KEYS-DIR/id_rsa.pub

   2. In your GitHub account, navigate to the IAC-REPO-NAME repository

   3. Click Settings > Deploy Keys.

   4. Click Add Deploy Key and paste in the SSH public key you copied. Choose a Title for the key.

   5. Select the check box "Allow write access"

   6. Click Save.

4. Navigate back to your Cloud Shell session

5. Create theknown_hosts file for GitHub. In your Cloud Shell session, run the command:

   ssh-keyscan github.com >> ~/.ssh/known_hosts
   

6. Create a Cloud Storage bucket in yourbuild project and upload your SSH keys and known_hosts file to it. Replace SSH-KEYS-DIR with the path to the directory where you generated the SSH keys.

   gcloud storage buckets create gs://github-keys-$BUILD_PROJECT_ID --project=${BUILD_PROJECT_ID} --location=us-central1
   
   gcloud storage cp SSH-KEYS-DIR/id_rsa* gs://github-keys-$BUILD_PROJECT_ID
   gcloud storage cp SSH-KEYS-DIR/known_hosts gs://github-keys-$BUILD_PROJECT_ID

7. Generate a Personal Access Token for GitHub. This token is used when performing Git operations using API calls that the recommender-parser service makes to generate pull requests, check-in updated IaC manifests.

   1. In your GitHub account, in the upper-right corner of any page, click your profile photo, then click Settings.

   2. In the left sidebar, click Developer settings.

   3. In the left sidebar, click Personal access tokens

   4. Click Generate new token.

   5. Give your token a descriptive name.

   6. Select the scopes as repo.

   7. Click Generate token.

   8. Copy the token to your clipboard.

   9. In your Cloud Shell session, create an environment variable.

      export GITHUB_PAT=personal-access-token-you-copied

Set up Cloud Build

1. Connect your IAC-REPO-NAME Git repository to integrate with Cloud Build.

   1. Go to the Cloud Build App page in the GitHub Marketplace.
   2. Scroll down and click Setup with Google Cloud Build at the bottom of the page.
   3. If prompted, Sign in to GitHub.
   4. Select Only select repositories. Use the Select repositories drop-down list to only enable access to your IAC-REPO-NAME in the Cloud Build app.
   5. Click Install.

   6. Sign in to Google Cloud.

      The Authorization page is displayed where you are asked to authorize the Google Cloud Build app to connect to Google Cloud.

   7. Click Authorize Google Cloud Build by GoogleCloudBuild. You are redirected to the Google Cloud console.

   8. Select your Google Cloud project.

   9. Enable the consent checkbox and click Next.

   10. In the Select repository page that appears, select the IAC-REPO-NAME GitHub repo

   11. Click Connect repository.

   12. Click Create a Trigger. This creates a trigger definition for you.

   13. Click Create to save your build trigger.

   For more information, see Running builds on GitHub.

2. The directory that you copied has a cloudbuild.yaml file. This configuration file outlines the steps that a Cloud Build job executes when triggered.

   steps:
   - name: hashicorp/terraform:0.12.0
     args: ['init']
   - name: hashicorp/terraform:0.12.0
     args: ['apply', '-auto-approve']
   

3. Add permissions to your Cloud Build service account to allow it to create service accounts, associate roles, and virtual machines (Compute Engine instances) in the test project

   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:$[e-mail] \
     --role roles/compute.admin \
     --project $TEST_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:$[e-mail] \
     --role roles/iam.serviceAccountAdmin \
     --project $TEST_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
   --member serviceAccount:$[e-mail] \
   --role roles/iam.securityAdmin \
   --project $TEST_PROJECT_ID
   

4. Open the Build Triggers page in the Google Cloud console.

5. Select build project, click Open.

6. Update the trigger's definition:

   1. Click menu and then click Edit.
   2. For Configuration, select the Cloud Build configuration file (yaml or json) option and type in cloudbuild.yaml in the text field.
   3. Click Save.

7. To manually test the build trigger, click Run on your trigger's entry in the triggers list.

8. Verify that a Compute Engine instance called tf-compute-1 and a service account called Terraform Recommender Test are created in your test project by the Cloud Build job you ran in the previous step

Note: You might need to wait a few moments for the Cloud Build job to complete before verifying the creation of resources in your test project.

Deploy the recommender-parser Cloud Run service

1. In Cloud Shell, Change directories to the directory created by cloning the repository

   cd $HOME/recommender-iac-pipeline-nodejs-tutorial/parser-service
   

2. Configure Google Cloud to use a default region for Cloud Run services. In this tutorial, you use the us-central1 region but you can choose a different supported region if you prefer.

   gcloud config set run/region us-central1
   

3. The parser-servicedirectory has a stub sub-directory which has a few sample payload JSONs for you to test the recommender-parser service with. Run the following sed commands to replace the PROJECT_ID placeholders in these JSONs with your test project ID.

   sed -i "s|__PROJECT_ID__|${TEST_PROJECT_ID}|g" ./stub/iam.json
   sed -i "s|__PROJECT_ID__|${TEST_PROJECT_ID}|g" ./stub/vm.json
   

4. Run the following command to create an environment variable for your Docker image.

   export IMAGE=gcr.io/$BUILD_PROJECT_ID/recommender-parser:1.0
   

5. Build the image and upload to Container Registry

   gcloud builds submit --tag $IMAGE .
   

6. Create a service account for the recommender-parser service to interact with other Google Cloud services in the pipeline. It is a good practice to grant granular permissions to your Cloud Run services, refer to Cloud Run service identity for more details.

   gcloud beta iam service-accounts create recommender-parser-sa \
     --description "Service account that the recommender-parser service uses to invoke other Google Cloud services" \
     --display-name "recommender-parser-sa" \
     --project $BUILD_PROJECT_ID
   

7. The recommender-parser service needs to access the GitHub SSH keys and Terraform state you uploaded to Cloud Storage buckets created earlier. Add the service account as a member to the Cloud Storage bucket.

   gcloud storage buckets add-iam-policy-binding gs://github-keys-$BUILD_PROJECT_ID \
     --member=serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role=roles/storage.objectUser
   
   gcloud storage buckets add-iam-policy-binding gs://recommender-tf-state-$BUILD_PROJECT_ID \
     --member=serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role=roles/storage.objectUser
   

8. Give the recommender-parser service's service account access to Firestore, Recommender, and the Service Usage API.

   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/datastore.user
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/recommender.iamAdmin
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/recommender.iamViewer
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/recommender.computeAdmin
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/serviceusage.serviceUsageConsumer
   

9. Deploy the Cloud Run service, which is called recommender-parser, by running the command. Replace GITHUB-ACCOUNT with your GitHub account username, not email. Accept any system prompts.

   gcloud run deploy \
    --image=${IMAGE} \
    --no-allow-unauthenticated \
    --region us-central1 \
    --platform managed \
    --service-account recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
    --set-env-vars="GITHUB_ACCOUNT=github.com:GITHUB-ACCOUNT,GITHUB_PAT=${GITHUB_PAT},SSH_KEYS_BUCKET=github-keys-${BUILD_PROJECT_ID},TERRAFORM_STATE_BUCKET=recommender-tf-state-$BUILD_PROJECT_ID" \
    --project $BUILD_PROJECT_ID \
    recommender-parser

Set up Firestore

1. In Google Cloud console, in your build project, navigate to the Firestore page.
2. When prompted for mode selection, click Select Native Mode.
3. Selectus-east1 as the default location.
4. Click Create Database.

Therecommender-parser service writes documents to this database for the following purposes:

• To keep track of the recommendations it has retrieved from the Recommender API
• Call the Recommender API once the recommendations are processed to update the status of each processed recommendation to SUCCEEDED or FAILED as appropriate. This is a key step that makes the pipeline idempotent by ensuring that recommendations are not processed incompletely or multiple times.

Set up a Cloud Scheduler job

1. Create a service account that Cloud Scheduler jobs use to run the recommender-parser service.

   gcloud beta iam service-accounts create recommender-scheduler-sa \
     --description "Service Account used by Cloud Scheduler to invoke the recommender-parser service" \
     --display-name "recommender-scheduler-sa" \
     --project $BUILD_PROJECT_ID
   

2. Give the service account run/invoker role to be able to invoke the Cloud Run service.

   gcloud beta run services add-iam-policy-binding recommender-parser \
   --member=serviceAccount:recommender-scheduler-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
   --role=roles/run.invoker \
   --region=us-central1
   

3. Get your recommender-service URL:

   gcloud beta run services list --platform managed --project $BUILD_PROJECT_ID
   

   Your Cloud Scheduler job invokes the recommender-parser service's /recommendation/iam route to parse IAM recommendations and the /recommender/vm route to parse VM sizing recommendations.

4. Create a variable for the endpoint that Cloud Scheduler jobs invoke. Replace RECOMMENDER-SERVICE-URL with the recommender-service URL you copied in the previous step.

   export RECOMMENDER_ROUTE_TO_INVOKE_IAM=RECOMMENDER-SERVICE-URL/recommendation/iam

   You URL should look like this sample URL after appending the route information:

   RECOMMENDER-SERVICE-URL/recommendation/iam
   

5. Create a Cloud Scheduler job called recommender-iam-scheduler.

   • Change the selected time-zone based on your location.
   • Replace IAC-REPO-NAME with the name of the GitHub repository you created.

   The message body takes three inputs and you must construct it as outlined below:

   • repo: This is the name of your GitHub repository IAC-REPO-NAME that you created in Create a GitHub repository.

   • projects: A list / array of Google Cloud projects IDs that this IaC GitHub repository maps to. In this tutorial, it is your test project.

   • stub: Recommender generates IAM recommendations when a subset of permissions for a role have not been used for 60 days and VM sizing recommendations follow a similar pattern. For the purposes of testing this pipeline on demand, stub can be passed in as true so that the pipeline is tested using the sample Recommender payloads provided in the repository that you cloned for this tutorial.

   gcloud beta scheduler jobs create http recommender-iam-scheduler \
     --project $BUILD_PROJECT_ID \
     --time-zone "America/Los_Angeles" \
     --schedule="0 */3 * * *" \
     --uri=$RECOMMENDER_ROUTE_TO_INVOKE_IAM \
     --description="Scheduler job to invoke recommendation pipeline" \
     --oidc-service-account-email="recommender-scheduler-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com" \
     --headers="Content-Type=application/json" \
     --http-method="POST" \
     --message-body="{ \"repo\": \"IAC-REPO-NAME\", \"projects\": [\"$TEST_PROJECT_ID\"], \"location\": \"global\", \"stub\": true }"

Note: --schedule="0 */3 * * *" runs the Scheduler job every 3 hours. Change it per your requirement. Also, if you need to create a scheduler job to parse VM sizing recommendations, you should append the your-recommender-parser service URL with /recommendation/vm and set the location attribute in the --message-body parameter to a valid zone.

Additional steps

Cloud Build publishes build information to a Pub/Sub topic called cloud-builds that was automatically created when you enabled the Cloud Build API in your build project.

1. Run the following command to verify the cloud-builds topic exists in your build project:

   gcloud pubsub topics describe cloud-builds
   

   If the topic exists, you will see the following output, where BUILD-PROJECT-ID is your build project ID:

   name: projects/BUILD-PROJECT-ID/topics/cloud-builds
   

   If you receive an error message saying the resource was not found, follow the instructions for subscribing to build notifications, to create the topic manually.

2. Create a service account that Pub/Sub uses to invoke the recommender-parser service endpoint.

   gcloud beta iam service-accounts create recommender-ci-subscription-sa \
     --description "Service Account used by Cloud Pub/Sub to push Cloud Build events to the recommender-parser service" \
     --display-name "recommender-ci-subscription-sa" \
     --project $BUILD_PROJECT_ID
   

3. The Pub/Sub service account should be associated with the roles it needs to be able to publish messages and invoke the recommender-parser service.

   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/pubsub.publisher \
     --project $BUILD_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/pubsub.subscriber \
     --project $BUILD_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/run.invoker \
     --project $BUILD_PROJECT_ID
   

4. Add the recommender-ci-subscription-saservice account you created to the recommender-parser service as a member with the invoker role

   gcloud beta run services add-iam-policy-binding recommender-parser \
     --member=serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role=roles/run.invoker --region=us-central1
   

5. Navigate to Pub/Sub in Google Cloud console.

6. Click the cloud-builds topic.

7. Click Create Subscription.

8. For Subscription ID, type recommender-service-build-events.

9. For Delivery Type, select Push.

10. For Endpoint, type in your recommender-service URL appended by /ci.

11. Check Enable Authentication.

    1. Select the service account recommender-ci-subscription-sathat you created.
    2. Click Grant in response to the prompt message.

12. Select Acknowledgement deadline as 60 seconds.

13. Keep rest of the defaults.

14. Click Create.

Test the pipeline

Recommender generates IAM recommendations when a subset of permissions for a role have not been used for 60 days. VM sizing recommendations follow a similar pattern. For the purposes of testing this pipeline on demand, you will use the sample recommendations JSON payloads provided in the stub sub-directory provided in the repository you cloned for this tutorial. This enables you to test the pipeline, barring the API calls that the recommender-parser makes to the Recommender API endpoint to update the status of recommendations it has successfully applied.

Alternatively, if you have active recommendations in your Google Cloud projects then you can test the pipeline end-to-end without needing to use stubs. The outcome outlined below is pertinent to when you are using the sample payloads for testing the pipeline. However, the steps to test this pipeline without samples remains the same.

1. In the Google Cloud console, navigate to your test project and review the resource that were created. You should have the following:

   1. A Compute Engine instance called tf-compute-1 with Machine Type g1-small.
   2. A service account named Terraform Recommender Test with the role of editor for your test project.

2. In the Cloud Scheduler console page in your build project, click Run now for the recommender-iam-scheduler job.

3. Click the job to view the logs. You can also view the recommender-parser service logs to get a detailed view of the steps being executed by the service.

4. Once the service completes its run, navigate to your GitHub IAC-REPO-NAME repository. The recommender-parserservice would have generated a pull request for you. Review the modified IaC manifests that comprise this pull request and click Merge Pull Request if you are satisfied with the changes to your IaC manifests.

   Warning: In production environments, we recommend that you enforce a robust approval workflow that includes human review of pull requests before changes are merged. The reviewers should have the knowledge to assess impacts identified in recommendations, as well as impacts specific to your infrastructure and business. Applying recommendations without proper assessment could result in unexpected changes, such as issues with system performance, poor reliability, or loss of required permissions. See the GitHub documentation for instructions to set up a review process.

5. A new commit to the master branch is created when you merge the pull request. This triggers a Cloud Build job that rolls out the modifications to the Google Cloud resources in your test project. Allow a few moments for the Cloud Build job to complete, you can review its status in the Google Cloud console

   Note: Making sure that you have a functional rollback strategy in place is a best practice when setting up your DevOps pipelines. For example, the infrastructure changes rolled out by the automation pipeline that you built in this tutorial can be rolled back by reverting to a previous Git commit. You can learn more about the https://git-scm.com/docs/git-reset and git revert commands in this Git documentation.

6. Once the job is complete, navigate to your test project. The sample payloads provided make the following changes to the resources in your test project.

   • Terraform Test service account which earlier has a role of editor when deployed, is changed to viewer.
   Note: When you run the Cloud Scheduler service to invoke parsing VM sizing recommendations at the /recommendation/vm endpoint then the Compute Engine instance tf-compute-1 which is originally has a Machine Type of g1-small is changed to f1-micro.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, delete both projects that you created.

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.
• Learn more about Google Cloud Policy Intelligence in the documentation.