Apache Airflow Documentation

What it is

Apache Airflow lets you define data pipelines as Python code: a DAG (directed acyclic graph) of tasks with dependencies, a schedule, retries and alerts. A scheduler runs the tasks at the right time, and a web UI shows what ran, what failed and why. The official documentation covers the core product, the Task SDK, the Docker image, the Helm chart for Kubernetes, the REST API and the provider packages that connect Airflow to AWS, Google Cloud, Azure, databases and many other systems.

Which version to learn

Airflow 3.0 was released in April 2025, and as of 2026-09 the latest release is 3.3.2. The Airflow 2 line reached end of life in April 2026. Airflow 3 changed a lot: a new UI, the Task SDK for writing DAGs, assets (formerly datasets) for event-driven scheduling, backfills managed by the scheduler, and DAG versioning. Learn Airflow 3, and be careful with tutorials and courses written before 2025. The managed services (Amazon MWAA and Google Cloud's managed Airflow) both offer Airflow 3 versions as of 2026-09.

How to use it well

• Start with the Quick Start to run Airflow locally (the Docker Compose setup is the easiest on most laptops).
• Read the Tutorial and the Core Concepts pages: DAGs, tasks, operators, the TaskFlow API, scheduling and the logical date.
• Then read the Best Practices page. It explains idempotent tasks, why top-level code in a DAG file slows the scheduler, and how to test DAGs.
• Use the provider documentation for your cloud when you connect to real services.

Prerequisites

Comfortable Python, basic Docker and a clear idea of what a pipeline does. Airflow is an orchestrator: it tells other systems when to run. Learn SQL and basic batch processing first, so you have something worth orchestrating.

Is it worth your time?

Yes. Airflow appears in a large share of data engineering job descriptions, and the managed versions on AWS and Google Cloud run the same open source code, so what you learn locally transfers directly. Alternatives such as Dagster, Prefect and Kestra are worth knowing about, but Airflow is the safest first orchestrator to learn.

Common questions

Is it free? Yes, Airflow is open source under the Apache 2.0 licence. Managed services charge for the infrastructure they run.

Should my tasks do heavy processing inside Airflow? Usually not. Keep Airflow for orchestration and push heavy work to a warehouse, Spark or a container, which scales better and keeps the scheduler healthy.