Data Pipeline Best Practices

Chapter 2 · Practice 4 of 4

Test that every DAG loads, and run full DAGs in staging before production

From Orchestration code hygiene

Why it matters

A DAG with an import error simply disappears from the scheduler. Airflow's docs recommend a basic loader test in CI that loads all DAGs and fails on any import error, plus unit tests for custom operators.

python
from airflow.models import DagBag

def test_no_import_errors():
    dagbag = DagBag(include_examples=False)
    assert dagbag.import_errors == {}

They also recommend a staging environment where full DAG runs are tested before production.

Be careful when removing tasks: the docs note that a deleted task no longer appears in the graph and grid views, which makes its history hard to find. If you need that history, create a new DAG instead of removing tasks from the old one.