Chapter 2 of 6 · 4 practices · 2 min · 4-question quiz
Orchestration code hygiene
The orchestrator parses your pipeline files constantly and runs tasks on different machines. These practices keep DAG files fast to load and tasks independent of where they run.
Start: Keep DAG files light: no queries, API calls or heavy imports at the top level4 practices in this chapter
- 1Keep DAG files light: no queries, API calls or heavy imports at the top levelAirflow parses DAG files repeatedly to schedule them. Any database query, network call or heavy computation at the top level of the file runs on every parse, not just when the task runs. Airflow's…
- 2Read variables and configuration inside tasks or through templatesReading an Airflow Variable at the top level of a DAG file triggers a database or network call on every parse. Airflow's docs recommend reading it at runtime instead: in templated fields, with…
- 3Pass small values between tasks and keep large data in shared storageTasks can run on different workers, especially with Kubernetes or Celery executors. Airflow's docs say not to store files or config on the local filesystem, since the next task may run somewhere else.
- 4Test that every DAG loads, and run full DAGs in staging before productionA DAG with an import error simply disappears from the scheduler. Airflow's docs recommend a basic loader test in CI that loads all DAGs and fails on any import error, plus unit tests for custom…
Check yourself on this chapter
4 questions, right here once you are signed in. Your progress is saved as you read.