Privacy-Preserving Beneficiary Deduplication for Relief NGOs After Floods

The problem: double distribution and too much shared data

After the 2022 floods in Pakistan, dozens of organisations registered affected families for food, cash and shelter. When lists are not compared, some families are registered three times while others are missed. When lists are compared, the usual method is to email spreadsheets of names, CNIC numbers and phone numbers between organisations, which exposes vulnerable people to fraud and misuse and may break the organisations' own data protection promises.

Relief coordinators face a choice between fairness and privacy. It should not be a choice.

The same problem returns with every disaster: heatwaves, earthquakes and the next monsoon. Organisations that cooperated once rarely have a safe, repeatable way to compare lists, so each response starts from spreadsheets and email again. A tool that makes safe comparison routine, and that each organisation can run without trusting the others with raw data, would change how coalitions work from the first week of a response, when mistakes in distribution are most common and most costly.

What already exists and the gap

• Splink, open source software from the UK Ministry of Justice, performs probabilistic record linkage at scale. It works on readable data held by one party.
• Large humanitarian agencies run their own registration systems, sometimes with biometrics, which smaller NGOs cannot use and which raise their own privacy concerns.
• Data collection tools such as KoboToolbox and ODK collect registrations well but do not deduplicate across organisations.

The gap is a lightweight tool that small and medium NGOs can run to find duplicates across organisations while each one keeps its readable data in-house.

Inspiration

Privacy-preserving record linkage using Bloom filters (Schnell, Bachteler and Reiher, BMC Medical Informatics and Decision Making, 2009) showed how to encode names as Bloom filters built from character pairs, so that similar names give similar encodings and can be compared without revealing the names. Later research has shown attacks on basic Bloom filter encodings and proposed hardening methods; your project should read that work and apply the defences that fit.

What you build

• Each NGO runs the encoder on its own computer. It cleans and normalises the data, then outputs encodings using a secret key agreed between the organisations. Nothing readable leaves the building.
• A coordinator (for example, a cluster lead) runs the matching service on the encodings and gets candidate pairs with similarity scores.
• Each NGO sees, for its own records only, which ones probably match a record elsewhere, and reviews them before acting.

The technically interesting part

• Name normalisation in two scripts. Names may be recorded in Urdu script or in many English spellings. Build a normalisation pipeline: transliteration rules, common variants, handling of honorifics and father's or husband's names, before encoding.
• Encoding design. Choose which fields go into which filter, the filter length, the number of hash functions and how to weight fields. Each choice trades accuracy against privacy.
• Hardening. Apply defences from the literature, such as keyed hashing, salting with a stable field and balancing, and document what each protects against.
• Scale. Comparing every record with every other is too slow; implement blocking, for example by district, that does not leak too much information itself.

How to evaluate it

• Match quality: precision and recall on a synthetic dataset built with realistic errors, compared with plain-text probabilistic matching on the same data.
• Privacy: attempt a frequency-based attack on your own encodings, as described in the literature, and report how many names you can recover with and without hardening.
• Performance: time to match two lists of 100,000 records each.
• Usability: a walkthrough with one or two NGO data officers.

Suggested plan

• Weeks 1 to 4: read the Schnell paper and later attack and defence papers; interview an NGO data officer.
• Weeks 5 to 10: synthetic data generator and name normalisation.
• Weeks 11 to 15: encoder, matcher and first quality results.
• Semester 2, weeks 1 to 6: hardening, attack experiments and blocking.
• Semester 2, weeks 7 to 12: review screens, packaging and NGO walkthrough.
• Semester 2, weeks 13 to 15: report and demo.

Path to a product

Humanitarian coordination bodies, NGOs and government social protection programmes all need deduplication with stronger privacy. An open source tool with paid setup and support for coalitions is realistic, and the project is directly relevant to privacy engineering and data roles.

Scope warning

Never use real beneficiary data during development; build a synthetic dataset. Do not claim the encodings are unbreakable; describe the threat model honestly. A duplicate match must never trigger automatic removal of aid; a person reviews every case.

Learn what you need on Sefism

• Data science roadmap: probability and evaluation behind record linkage.
• Cybersecurity roadmap: threat modelling and keyed hashing.
• pandas User Guide: cleaning and normalising the records.
• EU GDPR rules for business: a clear introduction to data minimisation and lawful sharing.
• Data engineering cheat sheet: building the matching pipeline.

Get help with this idea

The hard parts are realistic test data and an honest privacy analysis. Tauseef can help you plan the attack experiments, keep the scope manageable and present the trade-offs clearly. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session