Chapter 2 · Practice 3 of 4
Pass small values between tasks and keep large data in shared storage
Why it matters
Tasks can run on different workers, especially with Kubernetes or Celery executors. Airflow's docs say not to store files or config on the local filesystem, since the next task may run somewhere else.
| What to pass | How |
|---|---|
| A small value: a row count, a status, a path | XCom |
| A dataset, file or model | Remote storage such as object storage, with the path passed through XCom |
| Credentials | Connections, never XCom |
Treat XCom as a place for small messages between tasks, as the docs put it, not as a data store.