Public Procurement Red-Flag Analytics with Supplier Entity Resolution

The problem with reading public procurement data

Governments spend a large share of their budgets through public contracts, and many now publish tender and award data. Publishing is not the same as being usable. The same company appears as "ABC Traders", "A.B.C. Traders (Pvt) Ltd", "ABC Trading Co" and a misspelt version, sometimes with different addresses. Without resolving those into one supplier, you cannot answer basic questions: which firms win most often from one department, who always bids alone, which contracts were split just under the open-tender threshold.

Journalists, audit offices and civic groups do this by hand in spreadsheets, one investigation at a time, and the cleaning is thrown away afterwards.

What already exists and the gap

• Open Contracting Partnership publishes the OCDS standard, a data registry of published datasets, and a red flags guide with a long list of indicators.
• Cardinal is OCP's open-source command-line library that calculates red flags from OCDS data.
• OpenOpps and Tussell aggregate tender data commercially, mainly for suppliers hunting bids in Europe and the UK.
• Splink, from the UK Ministry of Justice, is an open-source Python package for probabilistic record linkage that runs on DuckDB or Spark.

The gap: the tools exist as parts. Cardinal assumes identifiers are clean; many publishers have weak or missing supplier IDs. There is no small, reproducible platform that resolves suppliers first, then computes indicators on the resolved entities, then explains each flag to a non-technical reader.

What you build

Three roles: public visitor (search and read), reviewer (confirms or rejects uncertain supplier matches), admin (manages sources and runs).

Flows:

1. Scheduled ingestion of OCDS releases and one messier source into a raw zone, then into cleaned staging tables.
2. Name normalisation (legal suffixes, punctuation, transliteration variants), then blocking and pairwise scoring.
3. Clustering matched pairs into supplier entities, with uncertain pairs sent to the review queue.
4. Indicator computation on resolved entities, stored with the evidence rows.
5. A web interface showing a buyer or supplier profile, contract timeline, flags and "why this flag" explanations.

The technically interesting part

Entity resolution at useful quality, with honest uncertainty. You will need blocking rules that keep candidate pairs manageable, comparison functions for names, addresses and tax or registration numbers where present, and an EM-trained Fellegi-Sunter model. Clustering matters too: if A matches B and B matches C, should A and C merge? Show how you avoid giant false clusters (for example by limiting cluster size or requiring a minimum edge weight).

The second hard part is defensible indicators. A red flag is a signal for further reading, not an accusation. Each indicator needs a written definition, the fields it depends on, what happens with missing data, and a false-positive discussion. The pattern resembles how Uber's Databook team describes attaching lineage and freshness to every dataset: every number on screen should link back to the rows behind it.

How to evaluate it

• Entity resolution: precision, recall and F1 on your 500+ pair gold set, at three thresholds; cluster-level metrics on a sample of 50 suppliers.
• Reviewer effort: how many pairs need human review per 10,000 records, and time per decision.
• Indicators: compare your results with Cardinal's output on the same OCDS data for the indicators both compute; explain differences.
• Performance: full rebuild time on the whole dataset; query latency for a supplier profile.
• Usability: three to five target users (a journalist, a student auditor, a civic volunteer) complete set tasks; report success rate and confusion points.

Suggested plan

1. Weeks 1 to 4: choose sources from the OCDS registry plus one local source, study the red flags guide, write indicator definitions.
2. Weeks 5 to 9: ingestion, staging models, normalisation rules.
3. Weeks 10 to 15: gold-set labelling, entity resolution model, review queue.
4. Weeks 16 to 20: indicators on resolved entities, evidence storage, API.
5. Weeks 21 to 25: web interface and methodology pages.
6. Weeks 26 to 30: evaluation, user sessions, write-up.

Path to a product

The first users are newsrooms and transparency NGOs, who can fund work through grants or pay for custom sources and training. A second route is compliance: companies and banks check suppliers and counterparties and would pay for an API on resolved entities. Keep the public site free and charge for bulk API access, alerts and hosted instances for audit offices.

Scope warning

Do not claim fraud or name individuals as wrongdoers; show flags and evidence only. Do not scrape sites that forbid it; start with officially published datasets. Do not try to cover every country: two sources done carefully beat ten done badly.

Learn what you need on Sefism

• Data Analytics roadmap : the SQL and modelling base the pipeline sits on.
• System Design cheat sheet : batch pipelines, storage and API design trade-offs.
• DuckDB Documentation : the engine Splink can run on locally.
• dbt Documentation : tested staging and indicator models.
• Fundamentals of Data Engineering : ingestion, lineage and serving layers.

Get help with this idea

The biggest risks are choosing sources with too little data and underestimating the labelling work for entity resolution. If you want a second opinion on scope, stack or how to evaluate it before you pitch it to your supervisor, a mentor can help you shape it. You still build it; the guidance is about making the plan realistic and the results defensible. New here? create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session