Which LLM Understands Roman Urdu and Bangla Business Messages? A Three-Model Evaluation Harness
Small businesses in Pakistan and Bangladesh get orders and complaints as WhatsApp-style messages in Roman Urdu, Urdu script, Bangla and English mixed together. Build an evaluation harness that compares three LLMs (for example one hosted frontier model, one cheaper hosted model and one open model) on real-shaped tasks such as extracting order details and classifying complaints, with a human-checked gold set, cost per correct answer and a public results page.
Intermediate
A small team, or one strong student willing to learn something new.1 semester · 2 students
6 to ship
3 optional extrasSuggested stack
What you should ship
- Three task definitions with JSON output schemas: order extraction (item, quantity, address, payment method), complaint classification, and reply drafting
- A gold dataset of at least 600 messages written by volunteers in Roman Urdu, Urdu, Bangla and code-switched English, each double-labelled by two people
- A harness that runs any model through LiteLLM or Ollama with fixed prompts, retries, timeouts and logging of tokens, latency and cost
- Scorers: exact and fuzzy field match for extraction, macro-F1 for classification, and a rubric-based LLM judge for replies calibrated against human ratings
- A results page per task and language with cost per correct answer and confidence intervals
- An error analysis chapter: spelling variants, script switching, numbers written as words, local address formats
If you have time left
- Adding a small fine-tuned open model and checking whether it beats the large models on cost per correct answer
- A leaderboard others can submit model results to
- Testing prompt variants (few-shot examples in Roman Urdu versus English instructions)
The problem: picking a model for a language the leaderboards ignore
A developer building an order bot for a Lahore clothing shop or a Dhaka grocery page has to choose a model. Public leaderboards measure English, maths and code. They say almost nothing about "2 suit chahiye, size M, COD, address DHA phase 5" or a Bangla complaint with English product names. The developer guesses, ships, and finds out from angry customers.
The questions are practical: which model extracts the right fields, which one handles Roman Urdu spelling chaos, and which one is cheapest per correct answer rather than cheapest per token?
What already exists and the gap
- General leaderboards such as LMArena and Hugging Face's Open LLM Leaderboard rank models on broad tasks.
- Multilingual benchmarks cover many languages but usually in standard script and formal text.
- Eval frameworks such as promptfoo, DeepEval and OpenAI Evals make it easy to run tests once you have a dataset.
The gap is the dataset and the method: informal, code-switched business messages with gold labels checked by two people, and a comparison that reports cost per correct answer, not only accuracy.
Inspiration
"Applied LLMs: What We Learned from a Year of Building" stresses that teams should build evaluations on their own data before choosing a model. Hamel Husain's and Shreya Shankar's writing on evals makes the same point about error analysis. Your project does exactly that for a language community that rarely gets its own benchmark.
What you build
- Tasks and schemas that mirror what a small business needs.
- A gold set written and labelled by real people, with agreement measured.
- A harness that runs three models and logs every call.
- A results page and an honest error analysis.
The technically interesting part
- Writing the data. Ask volunteers to write messages the way they really text, not the way they write essays. Never use real customers' messages.
- Fair comparison. Same prompt, same output schema, same retries. Report variance across repeated runs.
- Cost per correct answer. A model that is twice the price per token but makes half the mistakes may be cheaper for the business. Show it in one chart.
How you will evaluate it
- Inter-annotator agreement on the gold set (Cohen's kappa for labels).
- Per-task, per-language scores for all three models, with confidence intervals.
- Judge agreement with human ratings for the reply task.
Path to a product
The harness plus a small, well-labelled dataset is a consulting asset: businesses and agencies need "which model for our customers?" answered. A maintained public benchmark for South Asian business text could also attract sponsorship.
Scope warning
Three models and three tasks are enough. Do not fine-tune in the main scope. Keep model names and versions in every result, because providers change models over time.
Learn what you need on Sefism
- Choosing LLMs roadmap: how to compare models on quality, cost, latency and licence.
- AI evals cheat sheet: gold sets, scorers, judges and confidence intervals.
- LLM APIs cheat sheet: calling different providers consistently.
- LiteLLM documentation: one interface for many providers.
- AI evals: everything you need to know: the error-analysis-first method.
- Applied LLMs: what we learned from a year of building: why your own evals beat leaderboards.
- Ollama: run the open model locally.
- How do I choose which LLM to use for my project?: the decision your benchmark answers.
- What does an AI evals engineer do?: the job built around harnesses like this one.
Get help with this idea
The risk is a dataset too small or too clean to say anything. Tauseef can help you design the tasks, plan data collection and present results a developer can act on. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.