Cloud Composer Study Notes

Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow, an open-source platform for authoring and scheduling data pipelines. You write workflows as Python DAGs, and Composer handles the Airflow scheduler, web server, and worker infrastructure on Google Cloud. It coordinates tasks across services such as BigQuery, Dataflow, and Cloud Storage.

Key facts

• A DAG, or directed acyclic graph, is the Python definition of a workflow: it declares tasks, their dependencies, retries, and schedule in code that Airflow executes.
• Composer environments run on Google Kubernetes Engine infrastructure managed by Google, with the Airflow scheduler, web server, and workers deployed for you.
• DAG files are stored in a Cloud Storage bucket linked to the environment, and uploading a new DAG file makes it available to the Airflow scheduler automatically.
• Operators define what a task does: examples include BashOperator, PythonOperator, BigQueryInsertJobOperator, and operators for Dataflow, Dataproc, and Kubernetes.
• Composer supports Airflow features such as sensors that wait for a condition, XComs for small amounts of data passing between tasks, and connections for external systems.
• Scheduling uses cron-like expressions or Airflow presets, and a DAG run is created for each scheduled interval with its own execution date and task instances.
• Environment configuration, including size, software version, and PyPI packages, can be managed through the console, gcloud, Terraform, or the API.
• Composer integrates with Cloud Monitoring and Cloud Logging, so task failures, scheduler health, and environment metrics can trigger alerts and be investigated centrally.
• Cloud Composer 2 and later versions offer more flexible scaling, including worker autoscaling, compared with the earlier fixed-size Composer 1 architecture.

What the exam expects you to know

• The exam tests Composer as the orchestration answer when a workflow spans multiple services and needs dependencies, retries, and a schedule in one managed Airflow environment.
• Know that DAGs are Python code stored in Cloud Storage, and that changing a task usually means updating the DAG file rather than clicking in a console.
• Expect questions contrasting Composer with Cloud Scheduler: Scheduler triggers a single target on a cron, while Composer manages a full multi-step workflow graph.
• Candidates should recognise managed Airflow as the migration clue: existing Airflow DAGs move to Composer without adopting a proprietary workflow language.
• Understand environment sizing and upgrades at a high level, including that Airflow and Python package versions are properties of the environment.
• Questions may cover monitoring failed tasks through the Airflow UI, logs, and alerting, rather than treating Composer as a data-processing engine itself.

Cloud Composer vs Workflows: when to use which

Choose Cloud Composer when the team knows Apache Airflow, needs its rich operators and Python ecosystem, or is migrating existing DAGs that orchestrate complex data pipelines on schedules. Choose Workflows when you want a lightweight, serverless, YAML-defined orchestration of Google Cloud APIs and HTTP services with per-step pricing and no infrastructure to size. Composer runs an always-on environment suited to heavy data engineering schedules, while Workflows scales to zero for event-driven service coordination. The exam clue is Airflow or DAGs for Composer, and serverless step orchestration for Workflows. See also the Workflows notes.

Useful gcloud commands

gcloud composer environments list --locations=us-central1
gcloud composer environments run my-env --location=us-central1 dags trigger -- my_dag_id
gcloud composer environments storage dags import --environment=my-env --location=us-central1 --source=my_dag.py

Common exam traps

• Composer orchestrates tasks but does not itself process the data: the heavy work still happens in BigQuery, Dataflow, Dataproc, or another target service.
• A DAG should not contain heavy data processing in Python task code on the worker for large datasets; it should trigger the service designed for that processing.
• Cloud Scheduler alone cannot express task dependencies or branching between multiple steps, so multi-step dependency scenarios belong in Composer or Workflows.
• Deleting or pausing a DAG in the Airflow UI is not the same as deleting the Composer environment, which continues to incur environment costs.

Frequently asked questions

What is a DAG in Cloud Composer?

A DAG is a Python file that defines a workflow as a directed acyclic graph of tasks with dependencies, schedules, and retry behaviour. Composer's Airflow scheduler reads DAGs from the environment's Cloud Storage bucket, creates runs on schedule, and executes each task through its operator, recording state and logs for monitoring and troubleshooting.

How is Cloud Composer different from Cloud Scheduler?

Cloud Scheduler is a cron service that invokes one target, such as a Pub/Sub topic, HTTP endpoint, or App Engine app, at a set time. Cloud Composer manages entire workflows of many dependent tasks across services, with branching, retries, and backfills. Use Scheduler for a simple timed trigger and Composer when the trigger starts a coordinated pipeline.

Where are Composer logs and errors investigated?

Task and scheduler logs are available through the Airflow web interface linked from the Composer environment and are also sent to Cloud Logging for central search. Environment and task metrics appear in Cloud Monitoring, where alert policies can notify operators when DAG runs fail or tasks exceed expected duration, supporting a managed response process.

Study aid disclaimer: Cloud Cert Notes is an independent educational study aid. It is not affiliated with, endorsed by, or sponsored by Google, Microsoft, Amazon, Oracle, or Alibaba. Google Cloud, Associate Cloud Engineer, and Cloud Digital Leader are trademarks of Google LLC; all other certification names and product names are trademarks of their respective owners. Always confirm service details against the official provider documentation before sitting an exam.