Weakly Supervised Urdu and Roman Urdu Classification of Relief Needs During Floods

The problem: thousands of messages, no time to read them

During a major flood, requests for help pour in through social media posts, WhatsApp forwards, helpline transcripts and news reports. They are written in Urdu, Roman Urdu, English and mixtures, full of spelling variants and local place names. Someone needs rescue from a roof. A camp has run out of water. A family is looking for a missing child. Relief coordinators and volunteer groups cannot read everything, and the urgent messages get lost among prayers, news and rumours.

A classifier could sort messages into needs, but there is no time during a disaster to label thousands of examples, and models trained on English crisis data handle Roman Urdu poorly.

What coordinators do have is knowledge. They know the words people use when asking for rescue, the names of flooded villages and the phrases that usually mean a rumour. If that knowledge could be written down quickly as rules and turned into a trained model, the first useful classifier could exist on the first day rather than after the water has gone down.

What already exists and the gap

• Crisis informatics research has built classifiers for disaster tweets, and the Qatar Computing Research Institute's AIDR system combined crowd labelling and machine learning during disasters.
• Large language models can classify messages with no training, but at a cost per message, with variable accuracy in Roman Urdu and with data leaving the country.
• Volunteer groups sort messages by hand in shared spreadsheets.

The gap is a method suited to the first hours of a disaster in Pakistan: coordinators write quick rules in the language they know, a framework turns those noisy rules into training data, and a small model runs cheaply on everything, with every decision traceable.

Inspiration

"Snorkel: Rapid Training Data Creation with Weak Supervision" by Alexander Ratner, Stephen Bach, Henry Ehrenberg, Jason Fries, Sen Wu and Christopher Ré of Stanford, presented at VLDB 2018, showed that users can write labelling functions, small programs that label data or abstain, and that a generative model can combine their noisy votes into training labels good enough to train strong models without hand labelling. Your project tests that promise in a multilingual emergency setting.

What you build

• Relief coordinators see a map and list of needs by area, filter by type and open any message.
• Volunteers with local knowledge add or edit labelling rules in a simple interface and see their effect immediately.
• Researchers get a documented method and evaluation for Urdu and Roman Urdu crisis text.

The dashboard always shows the original message alongside the label and the rules that fired, so coordinators can judge for themselves. Nothing is acted on automatically.

The technically interesting part

• Labelling functions in three scripts. "Khana nahi", "کھانا نہیں" and "no food" mean the same. Combine keyword lists, transliteration-aware patterns and a zero-shot model as another voter.
• Combining noisy votes. Snorkel's label model estimates how accurate and correlated each function is without ground truth. Check its estimates against your test set.
• Place names. Village names have many spellings and repeat across districts. Matching them to a gazetteer, and saying when a match is ambiguous, is a hard and valuable part.
• Speed as a metric. The method's value is time. Measure quality against the hours spent writing rules versus labelling examples.

How to evaluate it

• F1 per need type on a hand-labelled test set, for weak supervision, a small hand-labelled training set of equal effort, and a zero-shot language model.
• Time to useful quality: F1 against hours of human effort.
• Place matching accuracy: share of messages placed in the right district or tehsil.
• Coordinator walkthrough: feedback from someone with relief experience on what they would trust.

Suggested plan

• Weeks 1 to 3: read the Snorkel paper, agree the label scheme, collect past flood messages within platform terms.
• Weeks 4 to 7: labelling functions, label model and classifier.
• Weeks 8 to 10: place extraction and the dashboard.
• Weeks 11 to 13: hand-labelled test set and the comparisons.
• Weeks 14 to 15: report and demo.

Path to a product

Disaster management authorities, humanitarian organisations and helplines need fast triage of incoming requests, and the same method works for heatwaves, earthquakes and disease outbreaks. An open toolkit with support during emergencies is realistic. The project is a strong NLP portfolio piece with clear social value.

Scope warning

Do not scrape platforms against their terms; use public data you are allowed to collect, past datasets and synthetic messages. Remove phone numbers and names before storing anything. This tool helps people triage; it must never decide on its own who gets help.

Learn what you need on Sefism

• Machine learning roadmap: training, evaluation and working with noisy labels.
• Data engineering roadmap: collecting and cleaning messages into a pipeline.
• Hugging Face Transformers Documentation: fine-tuning the multilingual classifier.
• spaCy Documentation: rule-based matching and entity extraction.
• Speech and Language Processing, 3rd Edition: text classification background.

Get help with this idea

The risk is spending the semester collecting data instead of testing the method. Tauseef can help you scope the data, the label scheme and the comparison. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session