Using serverless pipelines with Active Assist

Last reviewed 2021-06-09 UTC

This document is intended for enterprise architects and software developers who want to build an automation pipeline to use Active Assist in their Google Cloud organization. It's part of a series that discusses architectural patterns that enterprises can use to optimize their cloud footprint at scale using Active Assist. The series consists of the following parts:

Patterns for using Active Assist at scale
Using serverless pipelines with Active Assist (this document)
Using the GKE Enterprise toolchain with Active Assist

This tutorial shows you how to use Google Cloud serverless technologies to build an automation pipeline to retrieve and process Active Assist recommendations. You base the recommendations on the business rules set by your enterprise. The automation pipeline that you set up in this tutorial helps you to work with Active Assist at scale while still maintaining a team-led review and actuation process. This approach is useful when your enterprise wants to scale the use of the Active Assist portfolio but keep control of the review and actuation process within teams. It offers an alternative to using a continuous integration and continuous delivery (CI/CD) pipeline.

The architecture demonstrated in this tutorial is generic and you can extend it to work with other serverless products. The tutorial assumes that you are familiar with the following Google Cloud technologies:

To complete this tutorial, you must have an account for Slack or a similar notification or ticket processing tool. The tool must be set up on your machine and ready to use.

Architecture

Because the architecture demonstrated in this tutorial is modular, you can adapt the notifications component to match the requirements of your business. This tutorial demonstrates how to generate notifications and send them to Slack. You can also choose to send notifications to Pub/Sub or any other notification or ticket processing tool.

The following architectural diagram shows the components that you use in this tutorial:

Serverless pipeline.

The architecture has the following components:

A Cloud Run service that a scheduler triggers at fixed intervals. The service invokes the Recommender APIs by reading the metadata (project IDs and recommender types) that is defined and held in a Firestore collection.
A Pub/Sub topic that the Active Assist recommendations are pushed into and processed in.
A second Cloud Run service that parses Active Assist recommendations. This service determines how recommendations are processed based on the business rules defined by your enterprise and stored in a Firestore collection.
Two Firestore collections to store the business metadata and business rules. The Firestore collections operate as follows:
- The first collection stores the business metadata relevant to retrieving Active Assist recommendations. In this tutorial, recommendation type, Google Cloud project IDs, and locations attributes are used as business metadata. These attributes are used by the recommendation-collector Cloud Run service to determine which recommendation types are fetched.
- The second collection stores business rules that are applied when recommendations are processed.

Objectives

Create a sample Cloud Run service to retrieve Active Assist recommendations for a sample project and push them into a Pub/Sub topic.
Create two Firestore collections to store sample metadata and business rules respectively.
Create a second Cloud Run service to process recommendations according to the sample business rules that you define in this tutorial.
Create a Slack channel that the Cloud Run service sends sample Active Assist recommendations to.
Test the end-to-end pipeline with sample Active Assist recommendations.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

In the Google Cloud console, go to the project selector page.

Go to project selector
Select or create a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.
Make a note of the Google Cloud project ID for the recommendation manager project. You need this ID in the next section on setting up your environment.
Enable the Cloud Build, Firestore, App Engine,Pub/Sub, Cloud Run, Cloud Scheduler, and Cloud Source Repositories APIs.
Enable the APIs
You use the default application credentials for this tutorial. If you are prompted to create credentials on the Add credentials to your project page, click Cancel.
Make sure that billing is enabled for your Google Cloud project.
Create the following:
- A sample Slack channel.
- A sample Slack app and incoming webhook to receive notifications generated by an engine named recommendation-rules-engine. You set up the engine later in this tutorial.
For more information, see Create a Slack channel and Sending messages using Incoming Webhooks.

When you've created a Slack app and an incoming webhook URL, make a note of the URL as you need it later in this tutorial.

Building the serverless pipeline

In this section, you create the components that you need to build the serverless pipeline. The platform generates Active Assist recommendations based on usage patterns and system metrics. Depending on the recommendations that are generated, each recommendation category might use a different default time period from the past to analyze usage data and metrics.

If you have a sample Google Cloud project that has existing resources and Active Assist recommendations, you can run the pipeline to process those recommendations after you make the appropriate changes to the sample code provided.

Create the Firestore collections

In this section, you create two Firestore collections. The first, the activeassist-metadata collection, stores the business metadata relevant to retrieving Active Assist recommendations. The second, the activeassist-business-rules collection, stores the business rules that are applied when the pipeline processes recommendations.

When Active Assist recommendations are parsed, based on the business rules in the Firestore collection, either a notification is generated and sent, or the recommendation is automatically applied to the relevant Google Cloud resource.

Create the `activeassist-metadata` collection

In the Google Cloud console, go to the Firestore page.

Open Firestore
Create a Firestore database if you don't already have one. Alternatively, if you already have a Firestore database, skip to the next step.

Create the database:
1. Click Select Native mode to activate Firestore.
2. Select a region location close to the region where your Cloud Run services run.
3. Click Create Database. It takes a few moments to complete the configuration.
On the Firestore page, click Start Collection.
In the Collection ID field, enter the following: activeassist-metadata.

Populate the fields as shown in the following table. To add the next field, click Add Field.

Field name	Field type	Field value	Note
`project`	`string`	`Stub-Project-ID`	This tutorial uses a stub for the field value. If you want to use recommendations from an existing Google Cloud project, enter the project ID instead.
`locations`	`array`	`global`	Some recommendations might be region-specific or zone-specific, such as VM rightsizing recommendations. Other recommendations are global, for example, IAM recommendations.
`recommenderType`	`string`	`google.iam.policy.Recommender`	Not applicable.

When the fields are populated, click Save.

Create the `activeassist-business-rules` collection

Click Start Collection.

In the Collection ID field, enter the following: activeassist-business-rules

Populate the document as shown in the following table. To add the next field, click Add Field.

Field name	Field type	Field value	Note
`action`	`string`	`Notify`	Setting the value to `Apply` causes the service to apply the recommendation and remove the unused role.
`projectId`	`string`	`Stub-Project-ID`	This tutorial uses a stub recommendation. If you want to use recommendations from an existing Google Cloud project, enter the project ID instead.
`projectNumber`	`string`	`999999999`	This tutorial uses a stub recommendation. If you're using a recommendation from an existing Google Cloud project, enter the project number instead. You can find the project number in the Google Cloud console welcome page
`recommenderType`	`string`	`google.iam.policy.Recommender`	Not applicable.
`recommenderSubtype`	`string`	`REMOVE_ROLE`	Not applicable.
`slackWebhookURL`	`string`		Enter the Slack webhook URL that you generated in a previous step. The URL resembles the following: `https://hooks.slack.com/services/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`

This tutorial shows you how to construct a rule to determine if a recommendation is automatically applied or if a notification is generated and sent to a platform such as Slack. To learn how a recommendation can be automatically applied based on the evaluation of sample business rules that you set up, see the associated repository.

When the document is populated, click Save.

Creating a scheduled Cloud Run service

In this section, you create a scheduled Cloud Run service called recommendation-collector that invokes the Recommender API and retrieves active recommendations. The Identity and Access Management Recommender API is used in this tutorial as the Recommender API. The service reads metadata from the activeassist-metadata Firestore collection that you created to determine which recommendations to retrieve.

Click Open in Cloud Shell to open Cloud Shell for the recommendation manager project.

Open in Cloud Shell

When Cloud Shell opens, the following commands execute:
- The GitHub repository cloning command.
- The change directory command.
When the Open in Cloud Shell dialog appears, select Trust and then click Confirm.
Set the project ID and the project number of the current recommendation manager project as variables:
```
export RECO_MGR_PROJECT=PROJECT_ID
gcloud config set project $RECO_MGR_PROJECT
export RECO_MGR_PROJECT_NUMBER=$(gcloud projects describe $DEVSHELL_PROJECT_ID --format='value(projectNumber)')
```
Replace PROJECT_ID with your project ID. Once you have entered the commands, click Authorize when you're prompted.
Set the variable for the deployment region:
```
export REGION=us-central1
```
Although this tutorial uses the us-central1 region, you can use any region where Cloud Run is available.

Create an environment variable for your Docker image:

export RECOMMENDER_IMAGE=gcr.io/$RECO_MGR_PROJECT/recommendation-collector:1.0

Build the Docker image and upload it to Container Registry:
```
 gcloud builds submit --tag $RECOMMENDER_IMAGE
```
Create a service account for the recommendation-collector service to interact with other Google Cloud services in the pipeline:
```
gcloud iam service-accounts create recommendation-collector-sa \
  --description "Service Account that the recommendation-collector service uses to invoke other Google Cloud services" \
  --display-name "recommendation-collector-sa" \
  --project $RECO_MGR_PROJECT
```
It's a good practice to grant granular permissions to your Cloud Run services by assigning predefined roles to the service account. To learn more, see Service identity.

Give the service account for the recommendation-collector service access to Firestore and the Recommender API:

gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \
  --member serviceAccount:recommendation-collector-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/datastore.user
gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \
  --member serviceAccount:recommendation-collector-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/pubsub.publisher

If you're running this tutorial using the sample stub provided in the repository that you cloned, skip to the next step.

If you're building the pipeline in this tutorial using recommendations generated for an existing Google Cloud project, you must assign IAM permissions to the service accounts that you created to run the two Cloud Run services.

Set an environment variable, TEST_PROJECT_ID, with the ID of the project that you run this pipeline for before you execute the commands:
```
export TEST_PROJECT_ID=TEST_PROJECT_ID
gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
  --member serviceAccount:recommendation-collector-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/recommender.iamAdmin
gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
  --member serviceAccount:recommendation-collector-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/serviceusage.serviceUsageConsumer
gcloud services enable recommender.googleapis.com --project $TEST_PROJECT_ID
```
Make sure the Project ID you use matches the one you entered in creating the Firestore collections.

Note: If you are running this pipeline for recommender types that aren't used in this tutorial, make sure that you identify and assign the correct set of additional permissions. The service accounts that you create to run the two Cloud Run services need these permissions.

For this tutorial, you deploy the service with an environment variable called STUB_RECOMMENDATIONS. This variable lets you use a stub to test the pipeline.

Deploy the Cloud Run service:

gcloud run deploy recommendation-collector \
  --image=$RECOMMENDER_IMAGE \
  --no-allow-unauthenticated \
  --region $REGION \
  --platform managed \
  --service-account recommendation-collector-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --set-env-vars="STUB_RECOMMENDATIONS=true" \
  --project $RECO_MGR_PROJECT \

Accept any system prompts.

If you want to run the pipeline using Active Assist recommendations generated for a Google Cloud project, remove the following line from the command before you deploy it:

--set-env-vars="STUB_RECOMMENDATIONS=true"

Set up a Cloud Scheduler job to run the `recommender-collector service`

In Cloud Shell, create a service account for Cloud Scheduler jobs to use to run the recommendation-collector service:

gcloud iam service-accounts create recommender-scheduler-sa \
  --description "Service Account used by Cloud Scheduler to invoke the recommender-parser service" \
  --display-name "recommender-scheduler-sa" \
  --project $RECO_MGR_PROJECT

Give the service account the run/invoker role to enable it to invoke the Cloud Run service:

gcloud run services add-iam-policy-binding recommendation-collector \
  --member=serviceAccount:recommender-scheduler-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role=roles/run.invoker \
  --region=$REGION \
  --platform=managed

Get the recommendation-collector service URL:

export RECOMMENDER_SERVICE_URI=`gcloud run services describe recommendation-collector \
  --platform managed \
  --project $RECO_MGR_PROJECT \
  --region $REGION \
  --format="value(status.url)"`/run

Create a Cloud Scheduler job called recommender-iam-scheduler:
```
gcloud scheduler jobs create http recommender-iam-scheduler \
  --project $RECO_MGR_PROJECT \
  --time-zone "America/Los_Angeles" \
  --schedule="0 */3 * * *" \
  --uri=$RECOMMENDER_SERVICE_URI \
  --description="Scheduler job to invoke recommendation pipeline" \
  --oidc-service-account-email="recommender-scheduler-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com" \
  --headers="Content-Type=application/json" \
  --http-method="POST"
```
Set the time-zone to match your location. The time-zone value format is based on the tz database.

For more information, see gcloud scheduler jobs create http.

Your Cloud Scheduler job invokes the /run route for the recommendation-collector service.

Setting the --schedule="0 */3 * * *" flag runs the Scheduler job every three hours. You can change this setting according to your requirements. For more information, see Configuring cron job schedules.

Create the recommendation rules engine to process recommendations

In this section, you create a second Cloud Run service named recommendation-rules-engine to process recommendations that the recommendation-collector service collects. The recommendation-rules-engine service is invoked by Pub/Sub when new recommendations are pushed into the activeassist-recommendations topic.

This service parses recommendations based on the business rules that you defined in the activeassist-business-rules collection.

In Cloud Shell, open the recommendation-rules-engine directory:
```
cd ../recommendation-rules-engine
```
Note: If you get disconnected from your Cloud Shell session and need to reconnect, you can find the cloned repository directory at ~/cloudshell_open/activeassist-serverless-pipeline.

Create an environment variable for your Docker image:

export RULES_ENGINE_IMAGE=gcr.io/$RECO_MGR_PROJECT/recommendation-rules-engine:1.0

Build the Docker image and upload it to Container Registry:
```
gcloud builds submit --tag $RULES_ENGINE_IMAGE
```

Create a service account for the recommendation-rules-engine service to interact with other Google Cloud services in the pipeline:

gcloud iam service-accounts create recommendation-rules-sa \
 --description "Service Account that recommendation-rules-engine uses to invoke other Google Cloud services" \
 --display-name "recommendation-rules-sa" \
 --project $RECO_MGR_PROJECT

Give the service account for the recommendation-rules-engine service access to Firestore:

gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \
  --member serviceAccount:recommendation-rules-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/datastore.user

If you are using the stubs provided for this tutorial, proceed to the next step.

If you are testing the pipeline using recommendations generated for a Google Cloud project instead of the stubs provided for this tutorial, run the following commands to permit the rules engine service account access to your project:

gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
  --member serviceAccount:recommendation-rules-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/serviceusage.serviceUsageConsumer
gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
  --member serviceAccount:recommendation-rules-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/recommender.iamAdmin
gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
  --member serviceAccount:recommendation-rules-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/resourcemanager.projectIamAdmin

Deploy the Cloud Run service:

gcloud run deploy recommendation-rules-engine \
  --image=$RULES_ENGINE_IMAGE \
  --no-allow-unauthenticated \
  --region $REGION \
  --platform managed \
  --service-account recommendation-rules-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --project $RECO_MGR_PROJECT

Accept any system prompts.

Get the recommendation-rules-engine URL:

export RECOMMENDER_SERVICE_RULES_URI=`gcloud run services describe recommendation-rules-engine \
  --platform managed \
  --project $RECO_MGR_PROJECT \
  --region $REGION \
  --format="value(status.url)"`/process

The URL that you retrieve in this step is invoked when new recommendations become available by the Pub/Sub topic that you create in the next step.

Create a Pub/Sub topic for active recommendations

In this section, you create a Pub/Sub topic for the Active Assist recommendations that the recommender-collector service retrieves by invoking the Recommender API.

In Cloud Shell, create a Pub/Sub topic:

gcloud pubsub topics create activeassist-recommendations

Create a service account for Pub/Sub to use to invoke the recommendation-rules-engine Cloud Run service:

gcloud iam service-accounts create recommendation-engine-sub-sa \
  --description "Service Account used by Pub/Sub to push recommendations to the recommendation-rules-engine service" \
  --display-name "recommendation-engine-sub-sa" \
  --project $RECO_MGR_PROJECT

The Pub/Sub service account must be associated with the roles that it needs to publish messages and to invoke the recommendation-rules-engine service:

gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \
  --member serviceAccount:recommendation-engine-sub-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role roles/run.invoker \
  --project $RECO_MGR_PROJECT

Create a subscription for the Pub/Sub topic

Create a subscription for the recommendation-rules-engine service:

# grant Pub/Sub the permission to create tokens
PUBSUB_SERVICE_ACCOUNT="service-$RECO_MGR_PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud projects add-iam-policy-binding $RECO_MGR_PROJECT \
  --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
  --role='roles/iam.serviceAccountTokenCreator'
# configure the subscription push identity
gcloud pubsub subscriptions create active-assist-recommendations-for-rules-engine \
  --topic=activeassist-recommendations \
  --topic-project=$RECO_MGR_PROJECT \
  --push-auth-service-account=recommendation-engine-sub-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --ack-deadline=60 \
  --push-endpoint=$RECOMMENDER_SERVICE_RULES_URI

Allow the recommendation-engine-sub-sa service account that you created to invoke the recommendation-rules-engine service:

gcloud run services add-iam-policy-binding recommendation-rules-engine \
  --member=serviceAccount:recommendation-engine-sub-sa@$RECO_MGR_PROJECT.iam.gserviceaccount.com \
  --role=roles/run.invoker \
  --region=$REGION \
  --platform=managed

Running end-to-end tests using stubs

Active Assist recommendations are generated by the platform based on usage patterns and system metrics. Each recommendation category might use a different default window of time in the past to analyze usage data and metrics based on which recommendations are generated. For example, IAM recommendations are generated by the platform based on usage patterns from the past 90 days.

To test the end-to-end pipeline, the repository you cloned for this tutorial provides sample recommendations (stubs) that you use to run the end-to-end pipeline.

In this section, you do the following:

Inspect the stub recommendations.
Invoke the pipeline manually.
Check if a notification is generated and sent to the Slack channel that you created.

Review the sample recommendations provided in the repository:
```
cat ../recommendation-collector/stub.json
```
This file provides a sample recommendation with a REMOVE action for a sample role called roles/gkehub.connect.
Execute the following command to get Cloud Scheduler to run the job immediately, instead of waiting for the next scheduled run:
```
gcloud scheduler jobs run recommender-iam-scheduler
```
On the Cloud Scheduler console page, in the Result column for the recommender-iam-scheduler job, verify that the result is Success.

To get a detailed view of the steps that each of the services execute, you can also view the Cloud Run service logs for the recommendation-collector service and the recommendation-rules-engine service.
When the serverless end-to-end pipeline that you build in this tutorial runs successfully, it generates a Slack notification that contains details of the recommended role binding for you to remove. The following is an example of the notification that you receive:
```
Project xxxxxxxx\
**Impact**: SECURITY\
This role has not been used during the observation window.\
**Role**: roles/gkehub.connect\
**Member**:
serviceAccount:sample-sa@recommendation-sample.iam.gserviceaccount.com\
**Action**: remove
```

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

In the Google Cloud console, go to the Manage resources page.

Go to Manage resources

In the project list, select the project that you want to delete, and then click Delete.

In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

Learn about using Recommendations for Infrastructure as Code.
Learn more about Google Cloud serverless technologies.
Read about how to integrate Policy Intelligence recommendations into an IaC pipeline.