Inter-Annotator Agreement and Adjudication Tool for Data Labelling Teams
AI depends on labelled data, and labelling teams in Pakistan, India, Kenya and elsewhere produce a lot of it. Build a quality layer for labelling projects that measures agreement between annotators, spots unreliable labels and confusing guidelines, routes disagreements to an adjudicator, and estimates true labels from noisy votes, working with exports from Label Studio and similar tools.
Intermediate
A small team, or one strong student willing to learn something new.1 semester · 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- Importers for Label Studio exports and a simple CSV format, covering classification, multi-label and span labelling
- Agreement metrics: Cohen's kappa, Fleiss' kappa and Krippendorff's alpha, overall, per class and per annotator pair, with confidence intervals
- Per-annotator quality from gold questions and from agreement with consensus, shown fairly with sample sizes
- Label aggregation: majority vote compared with the Dawid-Skene model for estimating true labels
- An adjudication queue for high-disagreement items, with the guideline section and past decisions shown
- Guideline diagnostics: which classes and which guideline rules cause most disagreement
- A case study on a real labelling task you run with volunteers (for example Urdu sentiment or Roman Urdu intent) with at least 1,000 items and 5 annotators
If you have time left
- Active sampling: send items with the most uncertainty to more annotators
- Bounding box agreement for image labelling
- A summary report for clients of labelling vendors
The problem: labels are only as good as the people and the guidelines
Every supervised model, and every evaluation set for an LLM, rests on human labels. Labelling and AI training work has become a source of income for many people across South Asia and Africa. But quality is often checked crudely: a few spot checks, or "majority wins". Two careful annotators can disagree because the guideline is unclear, not because either is wrong, and that disagreement is useful information that usually gets thrown away.
What already exists and the gap
- Label Studio, Argilla, CVAT and doccano are open source labelling tools, with some agreement features in their paid or enterprise editions.
- Statistical packages compute kappa and alpha if you know how to prepare the data.
- Research on crowd labelling, starting with Dawid and Skene's 1979 model, shows how to estimate true labels and annotator reliability from noisy votes.
The gap is a free, focused quality layer that sits on top of existing labelling tools, explains agreement in plain language, separates "bad annotator" from "bad guideline", and supports a proper adjudication workflow.
Inspiration
The CrowdTruth project argued that disagreement between annotators is a signal about ambiguous data, not just noise. Your project builds tools that use that signal to improve both labels and guidelines.
What you build
- Importers and metrics.
- Annotator quality views that are fair to annotators.
- Aggregation and adjudication.
- Guideline diagnostics and a real case study.
The technically interesting part
- Choosing the right metric. Kappa and alpha handle chance agreement, missing labels and label types differently. Explain when each is appropriate.
- Fairness to annotators. An annotator who saw only hard items will look worse. Adjust for difficulty and show sample sizes, because these numbers can affect people's income.
- Dawid-Skene versus majority vote. Show when the model helps and when it does not, on your own data.
How you will evaluate it
- Agreement before and after a guideline revision informed by your diagnostics.
- Accuracy of majority vote versus Dawid-Skene against an expert-adjudicated subset.
- Feedback from annotators and an adjudicator on the tool's usefulness and fairness.
Path to a product
Labelling vendors need to prove quality to clients, and AI teams need to check vendors. A quality dashboard sold to vendors, or offered as an add-on to open source labelling tools, is a realistic product. The skills also lead into AI data work roles, which are growing.
Scope warning
Volunteers must consent and know how their work will be used. Do not publish individual annotators' scores. Keep to text classification and spans; leave images as a stretch goal.
Learn what you need on Sefism
- AI data work roadmap: labelling, guidelines, quality and AI trainer work.
- AI data work cheat sheet: agreement metrics and quality practices in one page.
- Label Studio: the labelling tool your importer reads.
- pandas user guide: the data handling behind the metrics.
- AI evals cheat sheet: why agreement matters for eval sets too.
- Data science roadmap: the statistics behind confidence intervals.
- Is data annotation a real career or just gig work?: what labelling teams do and where the work leads.
- What does an AI evals engineer do?: how agreement and labels feed AI evaluation.
Get help with this idea
The risk is a dashboard of numbers nobody knows how to act on. Tauseef can help you plan the case study and turn the metrics into clear decisions. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.