Data engineering

Stage 2 of 8 · 9 lessons · ~5 weeks

Analytical SQL and data modelling

Write the analytical SQL pipelines depend on, then design warehouse tables that answer business questions: facts, dimensions, grain, history and layers.

Start: Analytical SQL: CTEs and aggregation

Lessons· one lesson for each topic, in this order

  1. 1Analytical SQL: CTEs and aggregation
  2. 2Window functions
  3. 3SQL performance at scale
  4. 4OLTP and OLAP
  5. 5Facts, dimensions and grain
  6. 6Star and snowflake schemas
  7. 7Slowly changing dimensions
  8. 8Kimball, data vault and one big table
  9. 9Medallion and layered architecture

Do this

  • Solve 20 analytical SQL problems, including at least 8 with window functions.
  • Write a deduplication query that keeps the latest row per key with a deterministic tie-breaker.
  • Write the grain of your main fact table in one sentence.
  • Draw a star schema for your dataset with a date dimension and at least two others.
  • Implement one Type 2 dimension with valid_from, valid_to and a current flag.
  • Organise your tables into bronze, silver and gold schemas.

Start with

Practise

  • Fifty graded problems; the window function and aggregation ones match what pipeline SQL needs

  • An interactive book on one real dataset that ends with window functions and self joins

Keep handy

Also worth your time

Before moving on: A star schema design for your dataset: a written grain statement, a diagram of one fact table and at least three dimensions (one of them Type 2), and SQL that builds them in DuckDB or PostgreSQL with a deduplicated silver layer.

Tauseef's advice
Modelling is the skill that separates a data engineer from someone who moves files around. Always write the grain in one sentence before you create a fact table. If the SQL and databases roadmap is new to you, do its querying and joins stages first; this stage builds on them rather than repeating them.

Mark this stage as not done?