Data Pipeline Best Practices

Chapter 2 · Practice 2 of 4

Read variables and configuration inside tasks or through templates

From Orchestration code hygiene

Why it matters

Reading an Airflow Variable at the top level of a DAG file triggers a database or network call on every parse. Airflow's docs recommend reading it at runtime instead:

  • in templated fields, with {{ var.value.my_setting }};
  • or with Variable.get() inside a task function.

Do not hard-code environment-specific values in DAG code. Airflow recommends parameterising DAGs with environment variables, so the same code runs in staging and production. Keep credentials in Connections, not in task code.

Example

Python
from airflow.sdk import task

@task
def export(**context):
    from airflow.sdk import Variable  # read at runtime, not on every parse
    bucket = Variable.get("export_bucket")
    write_export(bucket, context["data_interval_start"])