Dataflow Study Notes

Dataflow is a fully managed service for running Apache Beam pipelines that process data in batch and streaming modes with the same code model. You define transforms in Java, Python, or Go, and Dataflow provisions workers, scales them, and handles shuffling and fault recovery. It is built for ETL, real-time analytics, and continuous data processing at scale.

Key facts

• Dataflow jobs run Apache Beam pipelines, so the same pipeline logic can process bounded data in batch mode and unbounded data in streaming mode.
• Workers are Compute Engine virtual machines managed by Dataflow, and the service can autoscale the number of workers up and down based on pipeline demand.
• Horizontal autoscaling and dynamic work rebalancing help streaming jobs redistribute work when data volume or key distribution changes during execution.
• Beam pipelines are built from PCollections, which are distributed datasets, and PTransforms, which are operations such as ParDo, GroupByKey, and Combine applied to them.
• Windowing divides a stream into logical windows based on event time, such as fixed, sliding, or session windows, so aggregations can be calculated over time ranges.
• Triggers control when window results are emitted, for example when a watermark passes the end of a window or after late data arrives.
• Dataflow templates package a pipeline for reuse, including classic templates and Flex Templates, so non-developers can launch jobs with parameters from the console or gcloud.
• Dataflow integrates directly with Pub/Sub for streaming input, BigQuery and Cloud Storage for output, and Bigtable for low-latency serving use cases.
• By default, Dataflow uses Compute Engine workers in your project, so VPC network, subnet, and service account choices affect how a job connects to sources.

What the exam expects you to know

• The exam tests Dataflow as the single answer for unified batch and streaming processing, especially when the same logic must handle both historical and live data.
• Expect scenario questions pairing Pub/Sub as the ingestion layer with Dataflow as the processing layer feeding BigQuery for analysis.
• Know that Dataflow is serverless and fully managed compared with self-managed Spark or Hadoop, including automatic scaling and worker management.
• Questions may cover templates as the way to let operators run standard pipelines without changing code, using runtime parameters for input and output.
• Understand windowing and event time at a conceptual level: late data, watermarks, and why streaming aggregation needs windows at all.
• Candidates should recognise Dataflow SQL and the Dataflow monitoring interface as tools for building and troubleshooting jobs without deep code changes.

Dataflow vs Dataproc: when to use which

Choose Dataflow when you want a fully managed, serverless pipeline, particularly for streaming, and you are willing to write or reuse Apache Beam logic. Choose Dataproc when you have existing Hadoop or Spark jobs to lift and shift, need specific open-source ecosystem tools, or want cluster-level control. Dataflow scales workers per job and bills for the job, while Dataproc runs persistent or ephemeral clusters you configure. For a new cloud-native ETL or real-time pipeline, the exam favours Dataflow; for migrating an on-premises Spark workload unchanged, it favours Dataproc. See also the Dataproc notes.

Useful gcloud commands

gcloud dataflow jobs list
gcloud dataflow jobs describe JOB_ID --region=us-central1
gcloud dataflow jobs cancel JOB_ID --region=us-central1

Common exam traps

• Dataflow is not a message queue or a warehouse: it processes data in motion or in files, but Pub/Sub delivers messages and BigQuery stores analytical results.
• Batch and streaming are not separate products in Dataflow: switching modes does not require rewriting the pipeline in a different framework like it would elsewhere.
• Leaving a streaming job running consumes resources continuously, so a forgotten streaming pipeline is a common cost trap compared with a batch job that ends.
• Autoscaling does not remove the need for efficient pipeline design: a skewed key in a GroupByKey can still create hot spots that slow the whole job.

Frequently asked questions

Can one Dataflow pipeline handle both batch and streaming data?

Yes. Apache Beam was designed so a single pipeline definition works with bounded sources such as files for batch and unbounded sources such as Pub/Sub for streaming. Dataflow runs the same transforms in either mode, changing how workers are managed and how results are produced, which is the main reason the exam highlights Dataflow for unified processing.

What are Dataflow templates used for?

Templates package a compiled pipeline so it can be launched repeatedly with different parameters, such as input topic or output table, without rebuilding code. Google provides ready-made templates for common jobs like Pub/Sub to BigQuery, and teams can create custom templates so operators, schedulers, or other services can start standard pipelines safely and consistently.

How does Dataflow handle late-arriving streaming data?

Streaming pipelines assign elements to windows based on event time and use a watermark to estimate when data for a window is complete. Triggers decide when to emit results, and allowed lateness settings let the pipeline accept data arriving after the watermark, updating results or sending very late elements to separate handling depending on the pipeline design.

Study aid disclaimer: Cloud Cert Notes is an independent educational study aid. It is not affiliated with, endorsed by, or sponsored by Google, Microsoft, Amazon, Oracle, or Alibaba. Google Cloud, Associate Cloud Engineer, and Cloud Digital Leader are trademarks of Google LLC; all other certification names and product names are trademarks of their respective owners. Always confirm service details against the official provider documentation before sitting an exam.