Dataproc Study Notes

Dataproc is a fully managed service for running Apache Hadoop, Spark, Hive, Pig, and related open-source data tools on Google Cloud clusters. It lets you create clusters quickly, run batch jobs, and delete clusters when work is done, rather than maintaining permanent on-premises Hadoop infrastructure. It suits migration of existing big data workloads with minimal code changes.

Key facts

• A Dataproc cluster consists of a master node that coordinates work and worker nodes that process data, with optional preemptible secondary workers for extra batch capacity.
• Clusters can be created in minutes and resized, and Dataproc autoscaling policies can add or remove workers based on YARN metrics for supported workloads.
• Dataproc separates compute from storage in typical designs by keeping data in Cloud Storage through the Cloud Storage connector, so clusters can be deleted without deleting data.
• Jobs can be submitted as Spark, PySpark, Hadoop, Hive, Pig, or Spark SQL, either through the console, gcloud, the REST API, or workflow templates.
• Workflow templates instantiate a cluster with a defined configuration, run a set of ordered jobs, and then delete the cluster automatically when finished.
• Component Gateway and optional components let you enable web interfaces such as Jupyter, Zeppelin, and HUE, plus tools like Presto or Flink, at cluster creation.
• Dataproc supports single-node clusters for small jobs and learning, standard clusters with one master, and high-availability clusters with three master nodes.
• Initialization actions are scripts run on nodes during cluster setup to install custom software or configure the environment before jobs start.
• By default at the time of writing, clusters use Compute Engine machines in a single zone, so region and zone choices affect availability and data locality.

What the exam expects you to know

• The exam tests Dataproc as the lift-and-shift answer for existing Hadoop and Spark workloads that should move to Google Cloud with minimal refactoring.
• Know ephemeral clusters: create a cluster for a job, store data in Cloud Storage, and delete the cluster afterwards to control costs.
• Expect questions on workflow templates for repeatable multi-step jobs and on autoscaling policies for variable batch workloads.
• Understand preemptible or spot capacity as a cost-saving option for fault-tolerant batch work, and why it is risky for the master node or urgent jobs.
• Candidates should distinguish Dataproc's cluster management responsibility from Dataflow's fully serverless job model in scenario comparisons.
• Questions may ask where data should live so clusters are disposable, with Cloud Storage as the correct answer rather than HDFS on worker disks.

Dataproc vs Dataflow: when to use which

Choose Dataproc when the team already has Spark, Hadoop, or Hive jobs, needs specific tools from that ecosystem, or wants to manage cluster shape and software versions directly. Choose Dataflow when building a new pipeline, especially a streaming one, and you prefer Google to manage workers, scaling, and shuffle with no cluster to maintain. The deciding clues are heritage and control: existing open-source jobs point to Dataproc, while new unified batch and stream processing in Beam points to Dataflow. Cost questions also differ: Dataproc charges for the cluster while it exists. See also the Dataflow notes.

Useful gcloud commands

gcloud dataproc clusters create my-cluster --region=us-central1
gcloud dataproc jobs submit pyspark gs://my-bucket/job.py --cluster=my-cluster --region=us-central1
gcloud dataproc clusters delete my-cluster --region=us-central1

Common exam traps

• Storing the only copy of data in HDFS on a temporary cluster risks losing it when the cluster is deleted, so persistent data belongs in Cloud Storage.
• Dataproc is not serverless in the same sense as Dataflow: you still choose machine types, node counts, and cluster lifetime, even though Google manages provisioning.
• A high-availability cluster protects the master role with three masters, but it does not make a single-zone deployment immune to a zone outage.
• Preemptible workers are cheap but can be reclaimed, so they suit fault-tolerant batch tasks and are a poor choice for critical, time-sensitive processing.

Frequently asked questions

Why store Dataproc data in Cloud Storage instead of HDFS?

Keeping data in Cloud Storage separates storage from compute, so you can create a cluster only for the duration of a job and delete it afterwards without losing input or output data. It also lets multiple clusters and other services access the same data. HDFS on cluster disks ties data lifetime to that specific cluster.

What is a Dataproc workflow template?

A workflow template defines a cluster configuration and an ordered list of jobs to run on it. When launched, Dataproc creates the cluster, runs the jobs in sequence, and deletes the cluster on completion. This makes recurring batch workflows repeatable, version-controlled, and cheaper than leaving a dedicated cluster running between runs.

When should I use autoscaling on a Dataproc cluster?

Use an autoscaling policy when batch workloads have variable demand and YARN has pending work that extra workers can usefully process. Autoscaling adds workers to shorten job time and removes them when idle to save cost. It works best for Spark and Hadoop batch jobs and is less useful for interactive or streaming workloads with steady needs.

Study aid disclaimer: Cloud Cert Notes is an independent educational study aid. It is not affiliated with, endorsed by, or sponsored by Google, Microsoft, Amazon, Oracle, or Alibaba. Google Cloud, Associate Cloud Engineer, and Cloud Digital Leader are trademarks of Google LLC; all other certification names and product names are trademarks of their respective owners. Always confirm service details against the official provider documentation before sitting an exam.