Roadmap · Specialise
Data engineering (Azure)
Opens with creditsFor students and early-career engineers who know some Python and SQL and want to build the pipelines that move, model and serve data, on AWS, Google Cloud or Azure. By the end you can design a warehouse model, run batch and streaming pipelines with Spark, Airflow, dbt and Kafka, deploy them on your cloud and prove they are tested, observable and affordable.
Curated by Tauseef Fayyaz8 stages, 63 topics~9 months at a few hours a weekBy the end, you will be able to
- Explain the data lifecycle from source to dashboard, and choose between batch and streaming, ETL and ELT, and a lake, warehouse or lakehouse for a given problem
- Write analytical SQL with window functions and design a star schema with a clear grain, slowly changing dimensions and medallion layers
- Process data in Parquet with Spark or DuckDB, partitioned so queries read only what they need
- Orchestrate idempotent pipelines in Airflow that retry safely and backfill cleanly, with dbt models that are tested and documented
- Build and run a pipeline on your cloud with its storage, warehouse, processing and streaming services, defined in Terraform and checked in CI
- Show a portfolio pipeline with data quality tests, lineage, PII handling and a cost note, and prepare for your cloud's data engineer certification
The path
8 stages, in the order to learn them. Open one to start.
1What data engineers doLearn the data lifecycle, the main pipeline patterns and the Python you need to pull data from files, databases and APIs reliably, with a local setup to practise on.8 topics · ~4 weeks
- 1The data engineer's jobA data engineer builds and runs the systems that move data from where it is created to where people use it.
- 2The data lifecycleAlmost every data platform follows the same data lifecycle, the path data takes from birth to use.
- 3Batch and streamingBatch processing handles data in chunks on a schedule.
- 4ETL and ELTETL stands for extract, transform, load.
- 5Ingestion: full loads, incremental loads and CDCThere are three common ways to copy data from a source.
- 6Python for pipelinesPython is the glue language of data engineering.
- 7Ingesting from APIsMany data sources are REST APIs, web addresses that return data when you ask.
- 8A local practice setupSet up a workspace on your own computer that you can break freely.
8 in this stageOpen this stage2Analytical SQL and data modellingWrite the analytical SQL pipelines depend on, then design warehouse tables that answer business questions: facts, dimensions, grain, history and layers.9 topics · ~5 weeks
- 1Analytical SQL: CTEs and aggregationMost SQL in data engineering is analytical.
- 2Window functionsWindow functions calculate a value for each row by looking at a group of related rows.
- 3SQL performance at scaleIn a warehouse, cost and speed depend mainly on how much data a query reads.
- 4OLTP and OLAPThere are two kinds of database work.
- 5Facts, dimensions and grainDimensional modelling is a way to design a warehouse, made popular by Ralph Kimball.
- 6Star and snowflake schemasA star schema puts one fact table in the middle, with dimension tables around it like the points of a star.
- 7Slowly changing dimensionsDetails in a dimension change over time.
- 8Kimball, data vault and one big tableThere are several styles for modelling a warehouse, and you should recognise them.
- 9Medallion and layered architectureMost platforms store data in layers, each cleaner than the one before.
9 in this stageOpen this stage- File formats, lakes and SparkStore data in the formats and layouts analytical engines expect, understand lakes, warehouses and lakehouses, and process large datasets with Spark.8 topics · ~6 weeksOpens with credits
- Orchestration with Airflow and transformation with dbtSchedule pipelines as Airflow DAGs that retry, backfill and alert safely, and build tested, documented SQL models with dbt.9 topics · ~6 weeksOpens with credits
- Your cloud data platformMove your pipeline to Azure: ADLS Gen2 and OneLake, Microsoft Fabric lakehouses and warehouses, Data Factory, Spark on Fabric or Azure Databricks, managed identities and a budget that protects you.7 topics · ~6 weeksOpens with credits
- Streaming basicsUnderstand event streams, Kafka's core ideas, delivery semantics and event time, and build a small streaming pipeline on Kafka and your cloud's streaming service.7 topics · ~4 weeksOpens with credits
- Quality, governance, cost and infrastructureMake pipelines trustworthy and repeatable: data tests and contracts, monitoring and lineage, PII handling, cost control, and Docker, Terraform and CI for your data code.9 topics · ~5 weeksOpens with credits
- Portfolio and your first data engineering roleTurn your pipeline into a portfolio project reviewers can follow, choose a certification if it helps, and apply for junior data engineering and adjacent roles with realistic expectations.6 topics · ~4 weeksOpens with credits
6 more stages, 46 more topics, free with an account.
You have seen the open stages of the Data engineering roadmap. Every topic it covers is listed above, and an account opens the rest: what each topic means, a question to check yourself, the courses, the practice and Tauseef's advice for every stage. Early access is open: sign up and your account is active shortly after.
Everything here is free to learn from. A few things, like adding a roadmap, keeping a cheat sheet or asking Tauseef a question, use credits. You get 15 when you join, your first roadmap is free, and if you are learning seriously and run out, you can ask for more. Credits only make sure the platform goes to people who are actually learning.
Related on Sefism
Picked from the other libraries by shared topics.
Best practices
See all- Data Pipeline Best PracticesHow to build data pipelines you can re-run, test and trust: idempotent tasks, orchestrator hygiene, layered transformations, data tests, freshness, ownership and monitoring.
- Data Analysis Best PracticesAn analyst's workflow in SQL, pandas and Polars: framing the question, reading data correctly, joins and grain, aggregation pitfalls, reproducible notebooks, checking results and sharing findings.
- Data Modeling Best PracticesDimensional data modeling for analytics: declaring the grain, fact and dimension tables, slowly changing dimensions, dbt marts, naming, tests, contracts and a semantic layer.
Checklists
See all- Data Quality ChecklistA data quality checklist for warehouse tables: ownership, freshness, volume, schema, nulls, uniqueness, referential checks, anomaly monitoring and data incidents, with how to verify each.
- Analytics Dashboard ChecklistAn analytics dashboard checklist for launch day: purpose and audience, metric definitions, data freshness, correctness, performance, access, accessibility, ownership and adoption.
- ML Model Deployment ChecklistAn ML model deployment checklist: baselines, training and serving consistency, validation, model registry, canary rollout, rollback, drift monitoring and ownership, with how to verify each.
Project structures
See all- dbt project project structureA dbt project that turns raw tables into tested, documented models, using the staging, intermediate and marts layers from dbt's own structure guide. It runs on DuckDB on your laptop, so you can learn analytics engineering for free, and grows into a governed project with checks, contracts, slim CI and a hosted docs site that moves to BigQuery or Snowflake unchanged.
- Apache Airflow project structureFor data engineers scheduling real pipelines with Airflow 3. It starts from
astro dev init(dags, include, plugins, tests and a Runtime Dockerfile that runs Airflow locally in Docker) and grows into dags grouped by business area, logic in a tested include package, asset-driven scheduling, data quality checks, alerts, CI that parses and tests every dag, and dag-only deploys. A self-hosted docker compose and Helm route is shown alongside. - GenAI RAG app (FastAPI) project structureFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
Career questions
See all- What is the difference between a data analyst, a data scientist and a machine learning engineer?Analysts answer business questions with SQL, spreadsheets, dashboards and reports. Data scientists add statistics, experiments and predictive models. ML engineers build and run models inside production software. Titles vary by company, so read the work in the job advert, and pick by the kind of problem you enjoy, not the title that sounds most senior.
- How do I switch from data analyst to data engineer?You already have the hardest part: SQL, business context and knowing what good data looks like. Add Python for pipelines, data modelling, an orchestrator, a cloud warehouse, Git and testing. Build one end-to-end pipeline, then move internally by taking on engineering work in your current team before applying outside.
- Can I get a remote data engineering job from Africa?Yes, it is realistic, and data engineering is one of the skills global employers most often hire remotely from Africa. You need solid SQL and Python, one cloud platform, a pipeline tool and a warehouse, two or three end-to-end projects, and a search aimed at remote-friendly companies and talent platforms. Most remote hires have some experience, so a local first job helps.
Cheat sheets
See allWork with me
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com