Pairwise Preference Labelling App for Urdu LLM Answers with Rater Calibration
Language models learn what 'good' means from human preference data, and very little of it is in Urdu. Build a mobile-friendly app where raters compare two model answers to an Urdu or Roman Urdu prompt, choose the better one against a clear rubric, and explain why. The system checks rater consistency, corrects for position bias, and produces a clean, documented preference dataset and a small leaderboard.
Intermediate
A small team, or one strong student willing to learn something new.1 to 2 semesters · 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- A prompt set of at least 300 Urdu and Roman Urdu prompts across everyday tasks, written by volunteers
- Answers from at least four models, collected through LiteLLM and stored with model and version
- A rating app with side-by-side answers, randomised order, a rubric (helpfulness, correctness, fluency, safety), ties and a short reason field
- Rater calibration: a training round with gold pairs, repeated pairs to measure self-consistency, and agreement between raters
- Position and length bias analysis, and model rankings using a Bradley-Terry model with confidence intervals
- A comparison of human preferences with an LLM judge's preferences on the same pairs
- A published dataset card and the dataset under a clear licence
If you have time left
- Bangla or Pashto prompts
- Training a small reward model on the collected preferences
- Voice prompts and spoken answers
The problem: whose idea of a good answer?
Modern chat models are tuned with human preference data: people compare two answers and pick the better one. InstructGPT (Ouyang and colleagues, 2022) made this approach famous. Most of that data is in English, rated by people with English-speaking norms. Urdu and Roman Urdu answers are often fluent-looking but awkward, too formal or culturally off, and there is little public data showing which models do better for Urdu speakers.
Preference labelling is also a growing kind of AI data work, and doing it well needs good tools and calibration.
What already exists and the gap
- LMArena (formerly Chatbot Arena) collects crowd preferences between anonymous models, mostly in English.
- Cohere's Aya initiative built multilingual instruction data with volunteers from many languages.
- Argilla and Label Studio support preference and ranking tasks.
The gap is a focused Urdu and Roman Urdu preference dataset collected with careful rater calibration and bias checks, plus open tools other language communities can reuse.
Inspiration
"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (Zheng and colleagues, 2023) documented position and verbosity biases in model judges. Humans have biases too. Measuring both on the same pairs is a strong research contribution.
What you build
- Prompts and answers from several models.
- A rating app that works well on a phone.
- Calibration, bias analysis and rankings.
- A published dataset with documentation.
The technically interesting part
- Rater quality. Repeated pairs show whether a rater is consistent; gold pairs show whether they follow the rubric. Use both, kindly.
- Bias. Randomise left and right, and measure whether longer answers win regardless of quality.
- Ranking. Bradley-Terry models turn pairwise wins into scores with uncertainty.
How you will evaluate it
- Inter-rater agreement and self-consistency.
- Size of position and length effects for humans and the LLM judge.
- Stability of rankings when you resample raters.
Path to a product
AI companies expanding into South Asia need local-language preference and evaluation data. A calibrated rater community with good tools can become a data services business, and the open dataset builds credibility.
Scope warning
Raters must consent and know the data will be published. Do not collect personal information in prompts. Keep the rubric short; four criteria are enough.
Learn what you need on Sefism
- AI data work roadmap: preference data, rubrics and rater quality.
- AI data work cheat sheet: the practices in brief.
- AI evals cheat sheet: judges, biases and how to check them.
- LiteLLM documentation: collecting answers from several models.
- Hugging Face Hub: publishing the dataset with a card.
- Choosing LLMs roadmap: how preference results feed model choice.
- Are AI trainer jobs worth it for graduates?: what rating work is like for the people doing it.
- Is data annotation a real career or just gig work?: where human data work leads.
Get help with this idea
The risk is a dataset too small or too noisy to say anything. Tauseef can help you plan rater recruitment, the rubric and the analysis. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.