Open Data Portal Quality Scorecard with Freshness and Broken Link Checks

The problem: open data that is not really open

Governments across South Asia have launched open data portals, and humanitarian groups publish thousands of datasets for researchers, journalists and developers. On paper, this is a huge public resource. In practice, students and analysts who try to use it hit the same walls: links that return errors, "datasets" that are scanned PDFs, files last updated years ago despite a promise of monthly updates, and no licence telling them whether they can reuse the data.

Portal teams rarely see the whole picture. They know how many datasets they host, not how many work. Users give up quietly. Nobody measures the problem regularly, so it never improves.

What already exists and the gap

• CKAN, the open source portal software behind many government portals, exposes a standard API for listing datasets and their files.
• The Open Data Barometer and similar global indices rank countries on open data policy and practice, using expert surveys rather than automated checks of every file.
• Link checkers for websites find broken links but know nothing about dataset metadata, formats or update promises.
• Open Data Portal Watch, a research project at the Vienna University of Economics and Business, monitored the metadata quality of hundreds of portals.

The gap is a regularly running, automated scorecard focused on South Asian and humanitarian portals, aimed at the people who can fix things, with results published openly.

Inspiration

Umbrich, Neumaier and Polleres' paper "Quality Assessment and Evolution of Open Data Portals" (FiCloud 2015) defined automated quality measures for open data portals, such as retrievability, usage of metadata fields, openness of formats and licences, and tracked them over time across many portals. A follow-up by Neumaier and Umbrich, "Measures for Assessing the Data Freshness in Open Data Portals" (2016), looked at whether datasets are updated as often as they claim. Your project applies these ideas to the portals that South Asian users actually depend on.

What you build

• The crawler visits each portal weekly, lists every dataset and records metadata, file links, formats, licences and update dates.
• The checker attempts to download each file (politely, with limits), records success, size and actual format, and compares the update date with the promised frequency.
• The scorecard summarises quality by portal and by publishing department, with trends.
• A portal team opens the worst offenders list and fixes the most-viewed broken datasets first.

The technically interesting part

• Different portals, one model. Map each portal's metadata to a common schema, as Portal Watch did. CKAN makes this easy; other portal software needs an adapter.
• Checking files politely. Use HEAD requests where possible, download only the first part of large files to detect format, respect rate limits and robots.txt, and run at quiet hours.
• Format truth. A file named ".csv" may really be an HTML error page; a ".xlsx" may be a scanned image. Detect real formats from file signatures and content, not names.
• Freshness. Compare last update dates with stated frequencies and flag datasets that are overdue, carefully handling missing or inconsistent dates.
• Fair scores. A department with five perfect datasets and one with five hundred mixed ones need different presentations. Show counts alongside percentages.

How to evaluate it

• Checker accuracy: manually inspect a random sample of 100 datasets and compare your judgement of downloadability and format with the tool's.
• Coverage: share of datasets on each portal the crawler reached.
• Stability: whether scores are consistent across weekly runs when nothing changed.
• Usefulness: share the worst offenders list with one portal team or civic tech group and record what they fixed or questioned.

Suggested plan

• Weeks 1 to 2: read the Portal Watch papers, choose portals, study their APIs.
• Weeks 3 to 6: crawlers and the common metadata schema.
• Weeks 7 to 9: file checks, format detection, freshness.
• Weeks 10 to 12: scorecard, trends and methods page.
• Weeks 13 to 14: manual validation, outreach to a portal team, report and demo.

Path to a product

Open data programmes are often funded by donors who want evidence of impact. A neutral, automated scorecard is useful to portal teams, funders and transparency groups, and could be offered as a monitoring service. For students, it is an excellent beginner data engineering project with real public value and a clear story to tell employers.

Scope warning

Do not try to judge the content quality of every dataset; stick to measurable properties. Never overload a portal; schedule runs gently and identify your crawler. Keep the tone of the scorecard constructive: the goal is fixes, not public shaming.

Learn what you need on Sefism

• Data engineering roadmap: scheduled pipelines, schemas and data quality checks.
• Data analytics cheat sheet: summarising results into a scorecard people read.
• Open Government Data (OGD) Platform India: one of the portals you could score.
• Pakistan Bureau of Statistics: an official publisher to compare against portal copies.
• pandas User Guide: cleaning and aggregating crawl results.

Get help with this idea

The risks are a crawler that breaks on every portal and scores nobody trusts. Tauseef can help you choose portals, define measures you can defend and present the results constructively. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session