Sefism Data Engineering Roadmap

What this roadmap is

The data engineering roadmap is Sefism's path for people who want to build the pipelines that move, model and serve data. It was written by Tauseef Fayyaz for students and early-career engineers who already know some Python and SQL, and it is built around one idea: pick a single dataset early and keep growing it, so that by the last stage you have one end-to-end portfolio pipeline instead of ten half-finished tutorials.

You choose a cloud at the start: AWS (the default), Google Cloud or Azure. Most of the roadmap is cloud-neutral and runs on your laptop with Python, DuckDB, PostgreSQL, Spark, Airflow, dbt and Kafka in Docker. The cloud platform, streaming and certification topics switch to your cloud's services.

By the end you should be able to:

• Explain the data lifecycle from source to dashboard, and choose between batch and streaming, ETL and ELT, and a lake, warehouse or lakehouse for a given problem.
• Write analytical SQL with window functions and design a star schema with a clear grain, slowly changing dimensions and medallion layers.
• Process data in Parquet with Spark or DuckDB, partitioned so queries read only what they need.
• Orchestrate idempotent pipelines in Airflow that retry safely and backfill cleanly, with dbt models that are tested and documented.
• Build and run a pipeline on your cloud, defined in Terraform and checked in CI.
• Show a portfolio pipeline with data quality tests, lineage, PII handling and a cost note, and prepare for your cloud's data engineer certification.

Who it suits, and who should start elsewhere

It suits computer science and software engineering students who like databases more than user interfaces, data analysts who want to own the pipelines behind their dashboards, and backend developers moving towards data. Tauseef's own advice in the last stage is that moving into data engineering from inside a data team is common and often easier than direct entry, so analyst and analytics engineer roles are worth applying for too.

If SQL joins and grouping are still new, do the querying stages of the SQL and databases roadmap first; this roadmap builds on them rather than repeating them. If you want to analyse data and present findings rather than build pipelines, the Data analytics roadmap fits better, and if statistics and models are the draw, look at Data science.

Prerequisites and realistic time

You need basic Python, basic SQL, a laptop that can run Docker, and a free cloud account with a budget alert. The roadmap adds up to about 200 hours across 8 stages:

• At about 5 hours a week (the default on the public page): roughly 40 weeks.
• At about 8 hours a week: roughly 29 weeks.
• At about 12 hours a week: roughly 21 weeks.

The first stage is a foundation stage: skipped if you say you are comfortable with the basics, and shortened to a review if you know some of it.

A sample week for a student (10 to 12 hours). Two evenings on the stage's topics and exercises, one evening on the checklist, and a weekend block extending your own pipeline to the stage's milestone. About five months of steady work.

A sample week for someone working full time (5 hours). Two short weekday sessions reading and running small examples, and one three-hour weekend block on your pipeline. Cloud work is best saved for these longer blocks, so you can switch everything off at the end of the session.

Stage by stage

1. What data engineers do (20 hours, foundation)

The data engineer's job, the data lifecycle, batch and streaming, ETL and ELT, full and incremental loads with change data capture, Python for pipelines, ingesting from APIs, and a local practice setup. Milestone: a Git repository with a Python script that pulls from a public API or database, saves raw data as dated files and loads a cleaned table into DuckDB or PostgreSQL, with logging and a README. The checklist asks you to rerun it for the same day and confirm the result is identical: the first test of idempotency.

2. Analytical SQL and data modelling (25 hours)

CTEs and aggregation, window functions, SQL performance at scale, OLTP and OLAP, facts, dimensions and grain, star and snowflake schemas, slowly changing dimensions, Kimball, data vault and one big table, and medallion layers. Milestone: a star schema for your dataset with a written grain statement, one fact table and at least three dimensions (one of them Type 2), built in SQL with a deduplicated silver layer. Tauseef calls modelling the skill that separates a data engineer from someone who moves files around.

3. File formats, lakes and Spark (30 hours)

CSV, JSON, Parquet and Avro, partitioning and file sizes, lakes, warehouses and lakehouses, open table formats (Iceberg and Delta Lake), how Spark works, PySpark DataFrames and Spark SQL, Spark performance basics, and designing a batch job. Milestone: a PySpark or DuckDB job that converts your raw data to date-partitioned Parquet, builds silver and gold tables, reruns for any date without duplicates, and a note comparing sizes and query times. You do not need a cluster: a laptop or a free managed notebook is enough.

4. Orchestration with Airflow and transformation with dbt (30 hours)

What an orchestrator does, Airflow DAGs, tasks and operators, schedules, data intervals and catchup, retries, idempotency and backfills, an optional look at Dagster and Prefect, and dbt models, sources, tests, documentation, incremental models and snapshots. Milestone: an Airflow DAG running locally in Docker that ingests a day of data and runs dbt build with tested staging, dimension and fact models, one incremental model and one snapshot, backfilled cleanly for at least seven past days.

5. Your cloud data platform (30 hours)

Account setup with budget alerts, identity for pipelines, then your cloud's services. On AWS: S3 and the Glue Data Catalog, Athena and Redshift, Glue jobs and EMR, MWAA and Step Functions. On Google Cloud: Cloud Storage, BigQuery, Dataflow and Dataproc, Cloud Composer. On Azure: ADLS Gen2 and OneLake, Microsoft Fabric (and where Synapse fits), Data Factory, and Spark on Fabric or Azure Databricks. Milestone: your pipeline on your cloud with least-privilege access, a budget alert and an architecture diagram with a measured weekly cost.

6. Streaming basics (20 hours)

Events, logs and brokers, Kafka topics, partitions and offsets, producers, consumers and consumer groups, delivery semantics, event time and windows, stream processing engines, and your cloud's streaming service (Kinesis, Firehose and MSK; Pub/Sub and Dataflow; or Event Hubs and Fabric Real-Time Intelligence). Milestone: a small streaming pipeline computing event-time windowed counts, with a note on duplicates and late events. The advice: learn streaming, but do not build everything as a stream.

7. Quality, governance, cost and infrastructure (25 hours)

Data quality checks, schema changes and data contracts, observability, freshness and alerting, lineage and catalogues, PII and access, cost control, Docker for pipelines, Terraform, and CI for data code. Milestone: data tests on every layer, a freshness alert, masked PII in gold, infrastructure in Terraform, a Docker image per job and a GitHub Actions workflow that lints, tests, parses DAGs and runs dbt build. This is the stage that makes a portfolio look like real work.

8. Portfolio and your first data engineering role (20 hours)

A portfolio pipeline that stands out, README and design notes, your cloud's certification (AWS Certified Data Engineer: Associate, Google Cloud Professional Data Engineer, or Microsoft Fabric Data Engineer Associate), entry routes, what hiring rounds test, and reading job descriptions. Milestone: a public repository, a CV that links to it, and 20 target roles with the skills they share.

Projects to build alongside

• Batch ELT Pipeline with Airflow, dbt and a Cloud Warehouse: the core portfolio project.
• Lakehouse with Medallion Layers and Data Quality Reports
• Real-Time Event Pipeline with Kafka and Windowed Metrics
• Public Budget Transparency Explorer: a smaller first end-to-end build on open data.

How it works on Sefism

Everything on Sefism is free to learn from. The public roadmap page lists every stage with its summary and length and opens the first two stages in full. Adding the roadmap to your dashboard opens every stage, lets you tick stages off and track progress, and gives you the companion cheat sheet. Your first roadmap is free to add; after that, adding a roadmap uses credits, and new members get starting credits when they join.

In the dashboard each topic has a short explanation, a check question and a prompt for an AI assistant that quizzes you instead of doing the work, and each stage has Tauseef's advice, a checklist and curated resources. If a stage leaves you stuck, you can ask Tauseef a question.

The companion cheat sheet

The Data engineering cheat sheet revises the same material in ten chapters, from ingestion and modelling to Spark, Airflow, dbt, streaming and governance, with cloud snippets in your edition.

Related on Sefism

• Data analyst to data engineer, on the most common route in.
• Which cloud to learn first: AWS, Azure or GCP and how to practise on the cloud without a surprise bill.
• Data Engineering Zoomcamp, dbt Learn and Kimball Dimensional Modeling Techniques, three resources the stages lean on.
• Column-Level Data Lineage and Impact Analysis from SQL Code, a harder project once you finish.
• Related roadmaps: Cloud engineering, DevOps and cloud and MLOps and LLMOps.

What you can do afterwards

Apply for junior data engineer, analytics engineer and data analyst roles with a pipeline you can explain end to end, including its tests, its failures and its cost. If you want to go deeper on infrastructure, the cloud engineering roadmap is a natural next step; if you want to serve models on top of your data, MLOps is.

FAQ

How long does it take to become a data engineer? The roadmap estimates about 200 hours: roughly 21 weeks at 12 hours a week, or about 40 weeks at 5 hours a week, plus the job search.

Do I need Spark to get a data engineering job? You should understand it, and stage three teaches it. Tauseef's note is honest about scale: many companies never need Spark, and explaining when DuckDB was enough shows judgement.

Which cloud is best for data engineering? The roadmap supports AWS, Google Cloud and Azure equally and advises one cloud deeply rather than three shallowly, because the concepts transfer.

Is a data engineering certification worth it? It can help you get noticed with no experience, but a working pipeline you can explain is what gets you hired. Book it only after finishing the free study path.

Is data engineering a good entry-level job? Direct junior roles exist, but the roadmap points out that moving in from an analyst or analytics engineer role is common and often easier.