Chapter 1 · Practice 3 of 4
Do not use the current time for anything that decides what data to process
Why it matters
Airflow's best practices say datetime.now() should never be used inside a task for critical computation. It is fine in log messages, but if a task uses it to decide which rows to load, a retry an hour later, or a backfill for last month, will process the wrong data.
The same applies to SQL: CURRENT_DATE or NOW() in a filter makes the query's result depend on when it ran.
| Instead of | Use |
|---|---|
datetime.now() - timedelta(days=1) | The run's data interval start and end |
WHERE created_at > NOW() - INTERVAL '1 day' | WHERE created_at >= :start AND created_at < :end |
Search your pipeline code for current-time calls and check each one.