Claim Verification Assistant with Source Citations
A claim verification assistant for newsroom fact-checkers and researchers that checks a factual claim against a defined, trusted corpus, such as official statistics, and returns supported, contradicted or not enough evidence, with the exact passages cited. An advanced NLP and RAG final year project that is honest about what automated fact checking can do.
Advanced
Needs a team, real planning, and a supervisor who knows the area.2 semesters · 2 to 3 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- An indexed evidence corpus from a defined source set, for example a statistics bureau's reports, with provenance and publication date stored per passage
- Claim decomposition that splits a compound statement into individually checkable parts
- Hybrid retrieval (keyword plus embeddings) with retrieval recall measured separately from verdict accuracy
- Stance classification labelling each passage as supporting, contradicting or unrelated, with special handling for numbers and dates
- A verdict of supported, contradicted or not enough evidence, always shown with the passages it rests on
- Evaluation on the FEVER dataset with three-way results compared to a published baseline
- A second evaluation on claims from outside the corpus, reporting how often the system correctly declines
If you have time left
- Temporal awareness so a claim true in one year and false later is judged against the right date
- A browser extension that checks a selected sentence against the corpus
- Confidence calibration reported as a reliability diagram
- Urdu claim support over an English corpus, evaluated on a small hand-built set
The problem fact-checkers face
Fact-checkers at newsrooms and civic organisations spend much of their time on a repetitive step: a politician or viral post states a number, and someone has to find the official report, the right table and the right year to see whether it holds. The claim spreads in minutes; the check takes hours.
Automated fact checking is frequently proposed and rarely scoped honestly. Deciding whether an arbitrary statement is true is not achievable. Checking a specific claim against a defined, trusted corpus, and reporting what that corpus says, is achievable, useful, and a good project.
What already exists and the gap
- Full Fact builds AI tools that help fact-checkers monitor media and spot claims worth checking.
- ClaimBuster, from the University of Texas at Arlington, scores sentences by how check-worthy they are.
- Google Fact Check Explorer searches fact checks that organisations have already published.
- The FEVER dataset is the standard research benchmark for verifying claims against Wikipedia, with supported, refuted and not enough info labels.
Most tools help find claims or find existing fact checks. The gap is the verification step against a specific trusted source, with evidence first and verdict second, and a system that says "not enough evidence" often and correctly. That is what a fact-checker can actually trust.
What you build
Users: a fact-checker who pastes a claim and reviews evidence, and an admin who manages the corpus.
- Corpus: a narrow, stated source set, such as national statistics reports or a ministry's published data, chunked with page, table and date.
- Pipeline: decompose the claim, retrieve candidate passages, classify stance per passage, aggregate a verdict.
- Interface: evidence passages highlighted and linked to the source page; verdict shown after the evidence.
The technically interesting part
The hard problem is stance classification. Deciding whether a passage supports or contradicts a claim is subtle, particularly with partial support, hedged language and numbers that nearly match. "Literacy is 62 percent" against a table saying 60.7 percent for one age group and a different year is neither a clean support nor a clean contradiction. Handle numbers and dates explicitly: extract them, normalise units and years, and compare them with rules before or alongside the model.
The second is making not enough evidence a first-class answer. Most claims a system like this receives will not be decidable from its corpus. A system that always says true or false is wrong most of the time in a way that is hard to notice.
How to evaluate it
- FEVER results: three-way label accuracy and FEVER score against a published baseline, so your numbers are comparable to research.
- Retrieval recall: does the right passage appear in the top 5? Report separately so errors can be located.
- Out-of-corpus test: at least 50 claims the corpus cannot answer; report the correct decline rate.
- Numeric claims: a hand-built set of 50 claims with near-miss numbers.
- Practitioner review: a working fact-checker or journalism lecturer rates 20 outputs for usefulness.
Suggested plan
- Weeks 1 to 4: choose the corpus, ingest and chunk it, set up FEVER locally.
- Weeks 5 to 9: retrieval (BM25, embeddings, hybrid) with recall measured.
- Weeks 10 to 14: stance classifier and verdict aggregation, FEVER evaluation.
- Semester 2, weeks 1 to 5: claim decomposition and numeric handling.
- Weeks 6 to 10: interface, out-of-corpus and numeric test sets.
- Weeks 11 to 14: practitioner review, report and demo.
Path to a product
The first users are fact-checking desks and research teams at civic organisations, who work with a small set of official sources. Offer the tool free to one organisation as a pilot, then charge newsrooms and think tanks for hosted corpora of the sources they care about, or seek grant funding, which is how much fact-checking infrastructure is paid for. Keep the verdict logic open so users can trust it.
Scope warning
- Choose the corpus narrowly and state it prominently. Checking claims against one organisation's published statistics is defensible. Checking anything is not.
- Do not build a social media monitoring system.
- Do not label claims as "fake news" publicly. The tool supports a human checker.
- Do not skip the out-of-corpus evaluation.
Learn what you need on Sefism
- AI engineering roadmap : retrieval, RAG and evaluation for grounded answers.
- AI engineering cheat sheet : embeddings, chunking and hybrid search at a glance.
- Google Research Blog : research on retrieval and factuality from a large lab.
- Our World in Data : an example of a trustworthy, well-sourced statistical corpus.
- Which AI topics are worth learning : where RAG skills fit in an AI career.
Get help with this idea
The two decisions that matter most are the corpus and how you evaluate declining to answer. Tauseef can help you narrow the corpus, plan the FEVER comparison and design test sets that show the system is honest, not just accurate. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.