Apache Airflow project structure

Apache Airflow project structure

Explorer

  • .git…
  • .astro
  • config.yaml
  • test_dag_integrity_default.py
  • dag_integrity_exceptions.txt
  • dags
  • .airflowignore
  • exampledag.py
  • include
  • assets.pynew
  • plugins
  • business_days_timetable.pynew
  • tests
  • conftest.pynew
  • .env, not committed
  • airflow_settings.yaml, not committed
  • Dockerfile
  • requirements.txt
  • packages.txt
  • .dockerignore
  • .gitignore
  • README.md
  • pyproject.tomlnew
  • .env.examplenew
  • Makefilenew
  • .sqlfluffnew
  • .vscodenew
  • settings.jsonnew
  • launch.jsonnew
  • .pre-commit-config.yamlnew
  • docker-compose.override.ymlnew
  • AGENTS.mdnew
  1. # Install the Astro CLI (macOS; see the docs for Linux and Windows)

    brew install astro
  2. # Create the project in an empty folder

    mkdir shop-pipelines && cd shop-pipelines && astro dev init
  3. # Start Airflow locally in Docker (UI on localhost:8080)

    astro dev start
  4. # Check every dag imports cleanly

    astro dev parse
  5. # Run the tests inside the Runtime image

    astro dev pytest
  6. # Run one dag end to end from the command line

    astro run orders_daily
  7. # Production: deploy dags only, or the full image when dependencies change

    astro deploy --dags
  8. # Self-hosted alternative: the official compose file

    curl -LfO https://airflow.apache.org/docs/apache-airflow/3.3.2/docker-compose.yaml