Project structure · Data and DevOps
Apache Airflow project structure
Opens with creditsFor data engineers scheduling real pipelines with Airflow 3. It starts from astro dev init (dags, include, plugins, tests and a Runtime Dockerfile that runs Airflow locally in Docker) and grows into dags grouped by business area, logic in a tested include package, asset-driven scheduling, data quality checks, alerts, CI that parses and tests every dag, and dag-only deploys. A self-hosted docker compose and Helm route is shown alongside.
Curated by Tauseef FayyazIntermediate139 files and foldersApache Airflow 3.3.2 (Task SDK 1.3), Astro Runtime 3.3-8, Astro CLI 1.46Checked October 2026How it grows
Each stage opens the explorer there
- 1Day 1What the scaffold and your first hour give you21 files and folders
- 2GrowingWhat a real app with features and tests adds+48 files and folders
- ProductionWhat running it for real users adds+35 files and foldersOpens with credits
- ExpertWhat a team needs to keep it healthy for years+35 files and foldersOpens with credits
How it is organised
- Dag files orchestrate; include/ does the work. A dag says when and in what order; the code that calls APIs, runs SQL and transforms data is plain Python in include/ with its own tests.
- No work at the top level of a dag file. The dag processor imports every file every few seconds, so a network call or heavy import there slows the whole deployment.
- Every task is idempotent and reads its time window from the run's data interval. Retries and backfills then produce the same result, never duplicates.
- Downstream dags are scheduled on assets, not on clock times, so they run when their input is actually ready.
- Connections and variables hold every secret and endpoint. Locally they come from .env and airflow_settings.yaml; in production from a secrets backend. Nothing secret is in Git.
- Use `airflow.sdk` imports (dag, task, Asset) in dag code. Tasks talk to Airflow through the Task SDK, not the metadata database directly.
- Every dag has an owner, area tags, retries, a timeout and a failure notifier, and a test enforces it.
- Group dags, SQL and tests by business area once there are more than a handful, so each team owns a folder.
At the top of the project
Open one in the explorer for the full story
- shop-pipelines/
- .git/Git history; run `git init` yourself after astro dev init.
- .astro/Astro CLI project files: project config and the dag integrity test it runs on parse.3 inside
- dags/Dag files: Python files the dag processor reads to find your pipelines.4 inside
- include/Everything your dags use that is not a dag: Python modules, SQL files, small reference files.5 inside
- plugins/Airflow plugins: things Airflow must register, such as timetables, listeners, macros or UI views.1 inside
- tests/pytest tests: dag checks that need Airflow, and unit tests for include code that do not.5 inside
- .envLocal environment variables: connections as AIRFLOW_CONN_*, variables as AIRFLOW_VAR_*, config as AIRFLOW__SECTION__KEY.
- airflow_settings.yamlLocal-only connections, pools and variables, loaded when you run astro dev start.
- DockerfileOne line at first: FROM the Astro Runtime image for your Airflow version.
- requirements.txtPython packages installed into the image: providers and your libraries, with pinned versions.
- packages.txtOperating system packages to install with apt, one per line.
- .dockerignoreKeeps .env, airflow_settings.yaml, logs and .venv out of the image.
- .gitignoreIgnores .env, airflow_settings.yaml, caches and local Astro files.
- README.mdThe template README: what each folder is for and how to start Airflow locally.
- pyproject.tomlSettings for ruff (including the AIR rules) and pytest, plus a dev dependency group.
- .env.exampleEvery connection and variable the project needs locally, with fake values.
- MakefileShort commands: make start, parse, test, lint, restart.
- .sqlfluffSQL lint settings for include/sql: Postgres dialect, Jinja templater, line length.
- .vscode/Shared editor settings.2 inside
- .pre-commit-config.yamlHooks on every commit: ruff, ruff format, sqlfluff for include/sql, a secrets scan.
- docker-compose.override.ymlExtra local services Astro starts alongside Airflow: a warehouse Postgres and MinIO as S3.
- AGENTS.mdInstructions for coding agents: airflow.sdk imports, no top-level work, logic in include/, run astro dev parse.
Learn more on Sefism
Roadmaps, cheat sheets and projects that go with Apache Airflow.
- Roadmap· AWSData engineering roadmapPipelines, warehouses, Spark, dbt
- Roadmap· Google CloudData engineering roadmapPipelines, warehouses, Spark, dbt
- RoadmapDevOps and cloud roadmapLinux, Docker, Kubernetes, AWS
- Cheat sheetData engineering cheat sheetPipelines, warehouses, Spark, dbt.
- Cheat sheetSQL and databases cheat sheetQueries, design, indexes, transactions.
- Cheat sheetDevOps and cloud cheat sheetLinux, Docker, Kubernetes, AWS.
- FYP idea· BeginnerOpen Data Portal Quality Scorecard with Freshness and Broken Link ChecksBuild a tool that crawls government and humanitarian open data portals every week, checks whether each dataset can actually be downloaded, opened and used, and publishes a scorecard of freshness, broken links, file formats and missing metadata. Many South Asian portals list thousands of datasets, and a large share turn out to be stale PDFs or dead links. The scorecard tells portal teams what to fix and tells users what to trust.
- FYP idea· IntermediateData Contract and Schema Drift Checker for Event PipelinesWhen an app team renames a field or changes a type, the analytics pipeline downstream breaks quietly and dashboards show wrong numbers for weeks. Build a tool that lets producers and consumers agree a data contract in a standard YAML format, checks every pull request and every batch of events against it, explains breaking changes before they ship, and tells the right people what will break.
- FYP idea· IntermediatePipeline That Turns Provincial Statistical PDFs into a Versioned Open DatasetGovernments publish important statistics as PDF yearbooks and reports: schools, hospitals, crops, prices and population by district. Build a data pipeline that extracts tables from these PDFs, cleans and checks them, tracks every change between editions, and publishes them as tidy, documented, versioned datasets that journalists, students and researchers can query directly.
See 8 moreShow fewer
- QuestionHow do I switch from data analyst to data engineer?You already have the hardest part: SQL, business context and knowing what good data looks like. Add Python for pipelines, data modelling, an orchestrator, a cloud warehouse, Git and testing. Build one end-to-end pipeline, then move internally by taking on engineering work in your current team before applying outside.
- QuestionCan I get a remote data engineering job from Africa?Yes, it is realistic, and data engineering is one of the skills global employers most often hire remotely from Africa. You need solid SQL and Python, one cloud platform, a pipeline tool and a warehouse, two or three end-to-end projects, and a search aimed at remote-friendly companies and talent platforms. Most remote hires have some experience, so a local first job helps.
- QuestionWhat is the difference between a data analyst, a data scientist and a machine learning engineer?Analysts answer business questions with SQL, spreadsheets, dashboards and reports. Data scientists add statistics, experiments and predictive models. ML engineers build and run models inside production software. Titles vary by company, so read the work in the job advert, and pick by the kind of problem you enjoy, not the title that sounds most senior.
- Resource· GuideApache Airflow DocumentationThe official documentation for Apache Airflow, the most widely used open source orchestrator for data pipelines. Start with the tutorial and the core concepts pages, and make sure you are reading the Airflow 3 docs, because many older tutorials use Airflow 2 syntax that has changed.
- Resource· RoadmapSefism Data Engineering RoadmapAn 8-stage path for students and early-career engineers who know some Python and SQL and want to build data pipelines on AWS, Google Cloud or Azure. It covers ingestion, analytical SQL and dimensional modelling, Parquet and Spark, Airflow and dbt, your cloud's data platform, Kafka streaming, data quality, Terraform and CI, and the first-role job search. You finish with one end-to-end portfolio pipeline that is tested, observable and costed.
- Resource· Cheat sheetSefism Data Engineering Cheat SheetThe revision companion to Sefism's data engineering roadmap: ten chapters on ingestion, analytical SQL and modelling, file formats and lakehouses, Spark, Airflow, dbt, cloud data platforms, streaming, data quality and infrastructure. Each concept gives what it is, why it matters and the classic mistake, with snippets where code helps. It comes in AWS, Google Cloud and Azure editions, and suits students and early-career engineers revising after each stage or before a technical round.
- Resource· GuideAmazon S3 User GuideThe official guide to Amazon S3, the object storage service most apps use for images, videos, uploads, backups and data files. Learn buckets, objects and keys, storage classes, access control and presigned URLs.
- Resource· Cheat sheetKubernetes Production Best PracticesLearnk8s's free, open production readiness checklist for Kubernetes workloads: 40 checks covering your application, manifests, security, scaling and going live. Each item explains why it matters and how to verify it. Use it before your first real deployment and as a review list for every service after.
Checked against 10 official sources, October 2026. If something has changed, the docs win.Show sourcesHide sources
- Apache Airflow documentation (3.3.2)
- Airflow: Running Airflow in Docker (docker-compose.yaml)
- Airflow: Best practices
- Airflow: Asset definitions and scheduling
- Helm chart for Apache Airflow
- Astro CLI: Develop your project
- Astro CLI source (scaffold templates)
- Astronomer deploy action
- Astronomer Cosmos
- Ruff: Airflow (AIR) rules
Best practices and checklists for this stack
Picked from the other libraries by shared topics.
Best practices
See all- Data Pipeline Best PracticesHow to build data pipelines you can re-run, test and trust: idempotent tasks, orchestrator hygiene, layered transformations, data tests, freshness, ownership and monitoring.
- Data Analysis Best PracticesAn analyst's workflow in SQL, pandas and Polars: framing the question, reading data correctly, joins and grain, aggregation pitfalls, reproducible notebooks, checking results and sharing findings.
- Data Modeling Best PracticesDimensional data modeling for analytics: declaring the grain, fact and dimension tables, slowly changing dimensions, dbt marts, naming, tests, contracts and a semantic layer.
Checklists
See all- Data Quality ChecklistA data quality checklist for warehouse tables: ownership, freshness, volume, schema, nulls, uniqueness, referential checks, anomaly monitoring and data incidents, with how to verify each.
- ML Model Deployment ChecklistAn ML model deployment checklist: baselines, training and serving consistency, validation, model registry, canary rollout, rollback, drift monitoring and ownership, with how to verify each.
- Analytics Dashboard ChecklistAn analytics dashboard checklist for launch day: purpose and audience, metric definitions, data freshness, correctness, performance, access, accessibility, ownership and adoption.
Work with me
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com