Public Budget Transparency Explorer

The problem with published budgets

Governments, city councils and public universities publish their budgets, and in practice they are close to unreadable: hundreds of pages of tables in PDF, with categories that change name between years so nothing can be compared. A journalist who wants to know how health spending in one province changed over five years spends days copying numbers into a spreadsheet. A citizen who wants to know what their city spends on waste collection gives up. The data is technically public and practically inaccessible.

This matters for accountability everywhere. In Pakistan, federal and provincial budget documents are published as PDFs every June, and the same pattern holds for many municipalities around the world.

What already exists and the gap

• OpenSpending, from Open Knowledge Foundation, is an open platform for publishing and exploring public spending data.
• USAspending.gov publishes US federal spending as searchable, downloadable data.
• The Open Budget Survey from the International Budget Partnership rates countries on budget transparency, not individual line items.
• Tabula and Camelot are open source tools for extracting tables from PDFs.

Where governments publish structured data, good explorers exist. Where they publish PDFs, the hard work of extraction and reconciliation is still manual, and tools stop at extracting one table. The gap is a cleaned, reconciled, multi-year dataset for one body, with an explorer on top and an honest account of its limits.

What you build

• Extraction: a pipeline that pulls tables from each year's budget PDFs into a normalised schema: year, level (head, sub-head, line item), category code, description, budgeted and revised amounts.
• Reconciliation: a mapping table that links categories across years when they are renamed, split or merged.
• Checks: every table's items must sum to its printed total; mismatches are logged, not silently fixed.
• Explorer: drill down from the top-level heads to line items, search for a keyword, see a trend over years.
• Comparison: planned versus revised or actual where both are published.
• Data quality report and downloads.

The technically interesting part

The real work is extraction and reconciliation, not visualisation. Budget PDFs have inconsistent table layouts, merged cells, footnotes that modify figures, numbers split across pages and totals that do not always add up. Show your supervisor your extraction accuracy on a sample checked by hand, the categories you had to map across years and how you decided, and the list of places where printed totals disagree with the sum of their parts. The cleaned dataset is more valuable than the interface on top of it.

How to evaluate it

• Extraction accuracy: hand-check a random sample of at least 200 line items against the PDFs and report the error rate.
• Reconciliation coverage: share of spending, by value, that maps cleanly across all years.
• Total checks: number of tables where items sum exactly to printed totals, and every exception explained.
• User test: give 5 people, ideally including a journalist or civic volunteer, three questions to answer with the explorer and time them.

Suggested plan

1. Weeks 1 to 2: choose the body and years, collect documents, design the schema.
2. Weeks 3 to 6: extraction pipeline and total checks.
3. Weeks 7 to 9: category mapping across years, data quality log.
4. Weeks 10 to 12: explorer interface and downloads.
5. Weeks 13 to 15: accuracy check, user test, data quality report.

Path to a product

Civic tech projects are usually funded rather than sold. Realistic paths are a partnership with a newsroom or a transparency NGO that publishes the explorer under its name, a small grant from a civic tech or open government programme, or paid data work for researchers who need clean budget series. Each new year's budget is a recurring piece of work, which keeps the project alive.

Scope warning

One government body or one municipality, several years. Do not attempt a national comparison; classification differences will consume the whole semester. Do not build commentary or judgement into the tool; present the data and its limits and let users draw conclusions.

Learn what you need on Sefism

• Data analytics roadmap : cleaning, SQL and visualisation in order.
• Frontend cheat sheet : building the explorer interface.
• PostgreSQL Documentation : schema, constraints and hierarchical queries.
• From Data to Viz : choosing charts for part-to-whole and trend questions.
• Storytelling with Data : making budget charts readable for non-experts.

Get help with this idea

The work is careful and plannable, which makes it a good team project. Tauseef can help you choose a body with usable documents, design the reconciliation approach and structure the data quality report. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session