Sefism Data Engineering Cheat Sheet

What this cheat sheet is

The data engineering cheat sheet is the revision companion to Sefism's Data engineering roadmap. It is written for students and early-career engineers and covers the same ground, from the data lifecycle and analytical SQL to modelling, Spark, Airflow, dbt, streaming and running pipelines on a cloud, as short entries you can reread in a minute.

Every concept follows the same pattern: what it is in plain words, why it is used or when it matters, and where it helps, an example, a snippet and the classic mistake. The idea is to revise an idea in a line or two, check a snippet and spot the mistake before it reaches production.

Three cloud editions

The sheet follows the roadmap's cloud choice, so it has three editions:

• AWS (the default): cloud snippets use S3, Kinesis and Terraform on AWS, and the platform chapter maps each idea to Glue, Athena, Redshift, EMR and MWAA.
• Google Cloud: snippets use Cloud Storage, Pub/Sub and Terraform on Google Cloud, and the platform chapter maps ideas to BigQuery, Dataflow, Dataproc and Composer.
• Azure: snippets use ADLS Gen2, Event Hubs and Terraform on Azure, and the platform chapter maps ideas to Microsoft Fabric, Data Factory and Azure Databricks.

The chapters and concepts are the same in every edition; what changes is the cloud code and the services named.

Chapter by chapter

The sheet has ten chapters and 77 concepts.

1. Foundations and ingestion (10 concepts)

The data engineer's role, the data engineering lifecycle, batch and streaming, ETL and ELT, full and incremental loads, change data capture, API ingestion with a snippet, Python for pipelines, DuckDB, and the idempotent pipeline: the idea the whole sheet keeps returning to.

2. Analytical SQL and data modelling (12 concepts)

Common table expressions, join fan-out, window functions, deduplication, OLTP and OLAP, fact and dimension tables, grain, star and snowflake schemas, surrogate keys, slowly changing dimensions, and Kimball, Inmon and data vault. The SQL entries carry snippets you can adapt directly.

3. File formats, lakes and lakehouses (11 concepts)

Columnar storage, Apache Parquet, Apache Avro, partitioning, partition pruning, the small files problem, data lakes, data warehouses, lakehouses, open table formats and the medallion architecture.

4. Batch processing with Spark (6 concepts)

Spark's driver, executors and partitions, the shuffle, the PySpark DataFrame, broadcast joins, data skew and partition overwrite: the few ideas that matter for writing batch output safely and quickly.

5. Orchestration with Airflow (6 concepts)

Orchestration, the DAG, Airflow DAGs and tasks in code, the data interval, retries, timeouts and alerts, and backfills.

6. Transformation with dbt (5 concepts)

dbt models, materialisations, dbt data tests, incremental models and snapshots, each with a short snippet.

7. Your cloud data platform (7 concepts)

Object storage, the data catalogue, the cloud warehouse, managed Spark and ETL services, least privilege for pipelines, secrets management and cost control. This is the chapter that changes most between editions.

8. Streaming basics (10 concepts)

The event log, Kafka topics, partitions and offsets, consumer groups, schema registries, delivery semantics, event time and processing time, windows, watermarks, stream processing engines, and your cloud's managed streaming service.

9. Data quality and governance (6 concepts)

Data quality checks, data contracts, schema evolution, data observability, data lineage and PII handling.

10. Infrastructure, CI and career (4 concepts)

Docker for data jobs, Terraform, CI for data pipelines and the cloud data engineer certifications that job postings mention.

Each chapter lists up to three library resources to go deeper with, such as Data Engineering Zoomcamp, Kimball Dimensional Modeling Techniques, Apache Spark Documentation and dbt Learn.

Rules of thumb and the glossary

Ten rules of thumb close the sheet. Among them: write the grain of every fact table in one sentence before writing SQL; make every job idempotent; pass the run date into jobs instead of reading the clock; keep raw data untouched in a bronze layer; check row counts before and after every join; choose batch by default and stream only when minutes of delay cost real money; and set a budget alert before creating any cloud data resource.

A glossary of 140 short terms sits alongside, and every concept joins it automatically, so the sheet doubles as a dictionary for the jargon in data engineering job descriptions.

How to use it with the roadmap

• After each stage, read the matching chapter and explain each concept aloud without looking. The roadmap's topics link straight to the concepts that cover them.
• While building your pipeline, keep the SQL, Spark and dbt chapters open. The pitfalls (join fan-out, data skew, the small files problem) are the ones that cost real hours.
• Before a technical round, reread modelling, idempotency and the batch-or-streaming trade-off. Being able to explain why a pipeline is safe to rerun is, in Tauseef's words in the roadmap, what gets people hired.

How it works on Sefism

Everything on Sefism is free to learn from. The public page shows every chapter's title and introduction and opens the first two chapters in full. Keeping the sheet uses credits and opens all ten chapters in the member area; new members get starting credits when they join. Adding the Data engineering roadmap to your dashboard (your first roadmap is free) brings this sheet with it.

Related on Sefism

• Data analyst to data engineer
• Batch ELT Pipeline with Airflow, dbt and a Cloud Warehouse, the project most of these concepts end up in.
• The SQL and databases cheat sheet for the SQL fundamentals underneath chapter two.
• The Cloud engineering cheat sheet for identity, networking and cost beyond the data services.

FAQ

What should a data engineering cheat sheet cover? This one covers the lifecycle, analytical SQL, modelling, file formats, Spark, Airflow, dbt, cloud platforms, streaming, data quality, Terraform and CI: the stages of the roadmap in the same order.

Is it enough to learn data engineering? No. It is for revising and checking. Learn through the roadmap's stages and resources, and use the sheet to remember.

Which edition should I choose? The cloud you are building on. If you have not decided, the concepts are identical; only the cloud snippets and service names change.

Does it help with certifications? It names the AWS, Google Cloud and Microsoft data engineer certifications and maps ideas to their services, but it is not an exam guide.