Lakehouse with Medallion Layers and Data Quality Reports

The problem with messy data at scale

Real datasets arrive messy: duplicate records, timestamps in the wrong time zone, trips with negative distances, and schemas that change from one month to the next. Most student projects clean a single CSV by hand in a notebook, and the cleaning cannot be rerun, audited or trusted. Companies of every size face the same thing with sales, sensor and app data, and the teams that cope are the ones who store raw data untouched, clean it in repeatable steps and measure quality every time.

The medallion pattern (bronze for raw, silver for cleaned, gold for business-ready) is now the common vocabulary for this, and it is a skill junior data engineers are expected to explain.

What already exists and the gap

• Databricks popularised the lakehouse and medallion layers on Delta Lake, with a free edition for learning.
• Snowflake and BigQuery offer managed warehouses that now read open table formats such as Iceberg.
• Great Expectations and Soda Core are open source tools for writing data quality checks as code.
• dbt is often used for the transformation layer on top of a warehouse.

The platforms are mature, and managed ones hide the interesting decisions. The gap for a student is a lakehouse built from open parts where every decision is visible: why this partition key, why this file size, why this row went to quarantine. That is also what a small company without a Databricks budget needs: an open, cheap setup that someone can maintain.

What you build

Pick one public dataset that updates monthly and has known issues. City taxi or ride trip records are a good fit; public transport, air quality or procurement data also work.

• Bronze: raw files exactly as downloaded, partitioned by source period, never edited.
• Silver: PySpark jobs that produce typed Delta or Iceberg tables, deduplicated with a deterministic rule, with rejected rows written to a quarantine table and a reason code.
• Gold: tables for two or three questions, for example trips and revenue per zone per day, or average trip time by hour and weekday.
• Quality checks: row counts, nulls, ranges, uniqueness and referential checks on each layer, stored as a results table and rendered as a report after every run.
• Table format features: MERGE for late corrections, schema evolution when a new month adds a column, and time travel to compare versions.

The technically interesting part

The hard part is making the pipeline idempotent and explainable at the same time. Rerunning March must produce the same silver table, not a second copy of March, which means MERGE on a deterministic key rather than append. Every rejected row needs a reason code so you can say "2 percent of rows failed the distance check, mostly in one month". Then there is the small files problem: partition too finely and a query opens thousands of tiny Parquet files; too coarsely and it scans everything. Show your supervisor measurements for at least two layouts and explain the choice.

How to evaluate it

• Quality findings: a table of every rule, how many rows failed per month, and what you think caused it.
• Idempotency: run each job three times and prove the row counts and checksums are identical.
• Performance: file count, average file size and query time for your gold queries before and after compaction and repartitioning.
• Correction: apply a late correction file and show the gold numbers change exactly where expected, with time travel for the before and after.
• Cost: hours of compute and storage used, and the monthly cost if it ran on a small cloud setup.

Suggested plan

1. Weeks 1 to 2: choose the dataset, profile it, design the folder and table layout.
2. Weeks 3 to 5: bronze ingestion and silver jobs with quarantine.
3. Weeks 6 to 8: gold tables and the quality check framework.
4. Weeks 9 to 11: MERGE, schema evolution, time travel, compaction experiments.
5. Weeks 12 to 15: report, cost estimate, demo.

Path to a product

For most students this is a portfolio piece for data engineering roles. It can also become a reusable open source starter ("a medallion lakehouse on open formats with quality reports built in"), or a consulting offer to small companies that have outgrown spreadsheets but cannot justify a managed platform. A realistic first client is a logistics or retail business with monthly exports and no data team; you would charge for setup and a monthly support retainer.

Scope warning

Do not build a dashboard product, a machine learning model or a custom orchestration system. Do not try several datasets. Use openly licensed data and run on a laptop, a free notebook environment or small cloud resources with auto-termination. Document every quality rule and why it exists.

Learn what you need on Sefism

• Data analytics roadmap : SQL and data modelling habits that the gold layer depends on.
• Apache Spark Documentation : DataFrames, partitioning and writing Parquet.
• Databricks Free Edition and Databricks Academy : a free place to practise Delta Lake and medallion layers.
• DuckDB Documentation : a fast second engine for checking your tables.
• Fundamentals of Data Engineering : the lifecycle and trade-offs behind each layer.

Get help with this idea

The trap is building layers without ever measuring quality or performance. Tauseef can help you choose a dataset with enough real problems, keep the scope finishable, and design the evaluation. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session