Data Pipeline Best Practices

Chapter 2 · Practice 1 of 4

Keep DAG files light: no queries, API calls or heavy imports at the top level

From Orchestration code hygiene

Why it matters

Airflow parses DAG files repeatedly to schedule them. Any database query, network call or heavy computation at the top level of the file runs on every parse, not just when the task runs. Airflow's docs give an example where an expensive top-level call adds 1,000 seconds to parsing.

Keep the top level to defining the DAG, its tasks and their dependencies. Move everything else inside task functions, including expensive imports, such as large data science libraries, that only one task needs.

To check, time how long a DAG file takes to load:

bash
time python dags/orders_daily.py

Run it a few times, since the first run may be slower, and subtract the bare interpreter start-up time.

Avoid

  • Querying the database at import time to decide which tasks to create