Chapter 1 of 6 · 4 practices · 3 min · 4-question quiz
Idempotent, re-runnable tasks
Every task will be retried, cleared or backfilled at some point. These practices make re-running a task produce the same result instead of duplicates or gaps.
Start: Make every task produce the same result when it runs again4 practices in this chapter
- 1Make every task produce the same result when it runs againAirflow's best practices say to treat tasks like database transactions: a re-run must give the same final result. The most common way this breaks is an
INSERTthat adds the same rows again on a… - 2Process a fixed time window from the run, never "the latest data"A task that reads "whatever arrived since last time" or "the latest file" gives a different answer each time it runs. Airflow's docs are explicit: never read the latest available data in a task; read…
- 3Do not use the current time for anything that decides what data to processAirflow's best practices say
datetime.now()should never be used inside a task for critical computation. It is fine in log messages, but if a task uses it to decide which rows to load, a retry an… - 4Plan backfills explicitly instead of relying on catchupIn Airflow 3, catchup is off by default (
catchup_by_default=False): the scheduler creates a run only for the latest interval. Turning catchup on makes the scheduler create a run for every missed…
Check yourself on this chapter
4 questions, right here once you are signed in. Your progress is saved as you read.