Data Contract and Schema Drift Checker for Event Pipelines
When an app team renames a field or changes a type, the analytics pipeline downstream breaks quietly and dashboards show wrong numbers for weeks. Build a tool that lets producers and consumers agree a data contract in a standard YAML format, checks every pull request and every batch of events against it, explains breaking changes before they ship, and tells the right people what will break.
Intermediate
A small team, or one strong student willing to learn something new.1 semester · 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- Contracts written in the Open Data Contract Standard for at least five event types of a demo app (for example an online store's orders, payments and deliveries)
- A pull request check that compares a changed contract or event schema with the main branch and classifies changes as safe, needs-migration or breaking
- A runtime checker on the event stream that validates samples against the contract and reports violations by producer and field
- Lineage from each contract field to the dbt models and dashboards that use it, so a breaking change lists exactly what it affects
- Notifications to the owning team with a suggested migration path (new version, dual writing, deprecation date)
- A demo with planted changes: a renamed field, a type change, a new required field and a silent unit change
- An evaluation of detection and false alarms on those changes
If you have time left
- Generating contracts from existing data by profiling
- Quality rules in the contract (ranges, freshness, uniqueness) checked on each run
- A web catalogue of contracts and owners
The problem: the app changed, the dashboard lied
A mobile team changes "amount" from rupees to paisa, or renames "city" to "city_name". Their tests pass. Downstream, the data pipeline keeps running, the revenue dashboard shows a hundredfold jump, and someone makes a decision on bad numbers. Data engineers spend much of their time chasing breakages like this.
Data contracts are agreements between the team that produces data and the teams that consume it: schema, meaning, quality rules and owners, written down and checked automatically.
What already exists and the gap
- The Open Data Contract Standard, maintained by the Bitol project under the Linux Foundation AI & Data, defines a YAML format for contracts.
- The Data Contract CLI validates and tests contracts.
- Schema registries such as Confluent's enforce compatibility rules for Kafka schemas.
- dbt model contracts and tools such as Great Expectations and Soda check data inside the warehouse.
The gap is connecting these in one workflow for a small team: catch the change in the producer's pull request, verify it at runtime, and show consumers exactly which models and dashboards will break, including semantic changes like units that schema checks miss.
Inspiration
API teams have used contract testing for years so that a backend change cannot silently break a mobile app. This project brings the same idea to data pipelines.
What you build
- Contracts for a demo app's events.
- A pull request check and a runtime checker.
- Lineage and notifications to the right owners.
The technically interesting part
- Classifying changes. Adding an optional field is safe; removing one is breaking; changing a unit is invisible to a schema. Design rules and quality checks that catch each kind.
- Lineage. Parse dbt's manifest to map fields to models and exposures.
- Runtime sampling. Validating every event is expensive. Sample smartly and still catch violations quickly.
How you will evaluate it
- Detection of each planted change, at pull request time and at runtime.
- False alarms on a set of safe changes.
- Time from a breaking change to the right team being notified.
Path to a product
Data teams in growing companies struggle with exactly this. A contract-checking GitHub App plus runtime monitor, or a consulting offer to set up contracts, are realistic paths, and the skills map directly to data and platform engineering roles.
Scope warning
Build on the standard and existing validators rather than inventing a new contract format. One demo app with five event types is enough.
Learn what you need on Sefism
- Data engineering roadmap: streaming, dbt, data quality and lineage.
- Data engineering cheat sheet: the patterns in brief.
- Apache Kafka documentation: the event stream your checker watches.
- dbt documentation: model contracts and the manifest for lineage.
- Fundamentals of Data Engineering: where contracts fit in the data lifecycle.
- GitHub Actions documentation: the pull request check.
- Can I get a remote data engineering job from Africa?: how a project like this fits a data engineering job search.
Get help with this idea
The risk is a tool that only catches the easy changes. Tauseef can help you design realistic planted changes and the rules to catch them. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.