Pipeline That Turns Provincial Statistical PDFs into a Versioned Open Dataset
Governments publish important statistics as PDF yearbooks and reports: schools, hospitals, crops, prices and population by district. Build a data pipeline that extracts tables from these PDFs, cleans and checks them, tracks every change between editions, and publishes them as tidy, documented, versioned datasets that journalists, students and researchers can query directly.
Intermediate
A small team, or one strong student willing to learn something new.1 to 2 semesters · 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- A catalogue of at least three recurring public statistical publications (for example a provincial development statistics yearbook) with source URLs, editions and licences
- Table extraction with per-table configuration, and a review screen to fix extraction errors
- dbt models that standardise district names, units and years, with tests for totals, ranges and missing districts
- Edition tracking: detect when a new edition revises an old number and record the change
- Published outputs as Parquet and CSV with a Frictionless-style data package description
- A small query page or notebook that answers five real questions with the cleaned data
- A data quality report per release
If you have time left
- OCR for scanned older editions
- A map view by district
- Alerts when a new edition is published on the source site
The problem: public data trapped in PDFs
National and provincial statistics offices across South Asia publish valuable data: the number of schools and teachers per district, hospital beds, crop areas, prices. Much of it comes as PDF yearbooks, sometimes scanned. A journalist or student who wants to compare districts over ten years must copy tables by hand, and every person repeats the same work.
When a new edition revises last year's numbers, nobody notices.
What already exists and the gap
- Open data portals such as Open Government Data Platform India publish machine-readable files for some datasets.
- Table extraction tools such as Camelot, Tabula and pdfplumber pull tables from PDFs.
- Our World in Data shows how clean, documented, versioned public data becomes widely used.
The gap is a maintained pipeline for specific recurring publications that are still PDF-only, with quality tests and edition tracking, so the cleaned data stays trustworthy as new editions arrive.
Inspiration
Data journalism teams often build exactly this kind of pipeline for a single story and then abandon it. The modern data stack (dbt, DuckDB, scheduled runs) makes it practical for a small team to keep it running.
What you build
- Extraction with per-table configuration and a review step.
- Transformation and tests in dbt.
- Versioned publishing with documentation.
- A query page that shows the data's value.
The technically interesting part
- Messy tables. Merged headers, totals rows, footnotes and page breaks. A configuration-driven extractor with a human review step is more reliable than full automation.
- Entity standardisation. District names change and are spelled differently across editions. Maintain a mapping table with history.
- Revisions. Detect and record when the same figure changes between editions.
How you will evaluate it
- Extraction accuracy against a hand-checked sample of tables.
- Number of dbt tests and how many issues they caught in the source data.
- Time to process a new edition end to end.
- Feedback from a journalist, researcher or teacher who uses the data.
Path to a product
Media organisations, NGOs and researchers need clean public data. A maintained open dataset can attract grants or sponsorship, and the pipeline skills map directly to data engineering roles.
Scope warning
Use only publicly available publications and record their licence or terms. Start with one publication and add more only when the first works end to end.
Learn what you need on Sefism
- Data engineering roadmap: pipelines, dbt, orchestration and data quality.
- Data engineering cheat sheet: the key tools and patterns in one page.
- dbt documentation: models, tests and documentation.
- DuckDB documentation: fast local analytics on Parquet.
- Apache Airflow documentation: scheduling the pipeline.
- Our World in Data: an example of documented public data done well.
- Data analyst to data engineer: how projects like this support a move into data engineering.
- Can I get a remote data engineering job from Africa?: how pipeline projects help a remote data engineering search.
Get help with this idea
The risk is spending the whole year fighting one PDF. Tauseef can help you choose publications, set realistic extraction goals and design the quality tests. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.