Reference patterns

This page provides links to sample code and technical reference guides for common Dataflow use cases. Use these resources to learn, identify best practices, and build features with sample code.

The reference patterns listed here are code-oriented and meant to get you quickly to implementation. To see a broader range of Dataflow solutions, review the list of Dataflow technical reference guides.

General analytics

Solution	Description	Links
Building a real-time website analytics dashboard	Learn how to build a dashboard that provides real-time metrics you can use to understand the performance of incentives or experiments on your website.	Sample code: Realtime Analytics using Dataflow and Memorystore Overview video: Level Up - Real-time analytics using Dataflow and Memorystore
Building a pipeline to transcribe and analyze speech files	Use Dataflow to transcribe and analyze uploaded speech files, then save that data to BigQuery for use in visualizations.	Sample code: Speech Analysis Framework

Solution

Description

Links

Building a real-time website analytics dashboard

Learn how to build a dashboard that provides real-time metrics you can use to understand the performance of incentives or experiments on your website.

Sample code: Realtime Analytics using Dataflow and Memorystore

Overview video: Level Up - Real-time analytics using Dataflow and Memorystore

Building a pipeline to transcribe and analyze speech files

Use Dataflow to transcribe and analyze uploaded speech files, then save that data to BigQuery for use in visualizations.

Sample code: Speech Analysis Framework

Log analytics

Solution	Description	Links
Building a pipeline to capture Dialogflow interactions	Use Dataflow to build a pipeline to capture and store Dialogflow interactions for further analysis.	Sample code: Dialogflow log parser

Pattern recognition

Solution Description Links

Solution	Description	Links
Detecting objects in video clips	This solution shows you how to build a real-time video clip analytics solution for object tracking by using Dataflow and the Video Intelligence API, allowing you to analyze large volumes of unstructured data in near real time.	Sample code: Video Analytics Solution Using Dataflow and the Video Intelligence API Apache Beam `Ptransform` for calling Video Intelligence API: apache_beam.ml.gcp.videointelligenceml module
Anonymize (de-identify) and re-identify PII data in your smart analytics pipeline	This series of solutions shows you how to use Dataflow, Sensitive Data Protection, BigQuery, and Pub/Sub to de-identify and re-identify personally identifiable information (PII) in a sample dataset.	Technical reference guides: De-identification and re-identification of PII in large-scale datasets using Sensitive Data Protection Sample code: Migrate Sensitive Data in BigQuery Using Dataflow and Cloud Data Loss Prevention

Detecting objects in video clips

This solution shows you how to build a real-time video clip analytics solution for object tracking by using Dataflow and the Video Intelligence API, allowing you to analyze large volumes of unstructured data in near real time.

Sample code: Video Analytics Solution Using Dataflow and the Video Intelligence API

Apache Beam Ptransform for calling Video Intelligence API: apache_beam.ml.gcp.videointelligenceml module

Anonymize (de-identify) and re-identify PII data in your smart analytics pipeline

This series of solutions shows you how to use Dataflow, Sensitive Data Protection, BigQuery, and Pub/Sub to de-identify and re-identify personally identifiable information (PII) in a sample dataset.

Technical reference guides:

De-identification and re-identification of PII in large-scale datasets using Sensitive Data Protection

Sample code: Migrate Sensitive Data in BigQuery Using Dataflow and Cloud Data Loss Prevention

Predictive forecasting

Solution	Description	Links
Predict mechanical failures using a vision analytics pipeline	This solution guides you through building a Dataflow pipeline to derive insights from large-scale image files stored in a Cloud Storage bucket. Automated visual inspection can help meet manufacturing goals, such as improving quality control processes or monitoring worker safety, while reducing costs.	Sample code: Vision Analytics Solution Using Dataflow and Cloud Vision API