Real-Time Sign Language Recognition from a Webcam
A real-time sign language recognition system for deaf signers and the people learning to communicate with them, recognising a working vocabulary of isolated signs, for example in Pakistan Sign Language, from an ordinary webcam. This computer vision final year project uses pose landmarks rather than raw video and reports accuracy on signers it has never seen.
Advanced
Needs a team, real planning, and a supervisor who knows the area.2 semesters · 2 to 3 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- A recorded dataset of at least 50 signs from at least 8 signers, with varied lighting and backgrounds, collected with written consent
- A landmark pipeline producing normalised hand, face and upper-body keypoints per frame, with handling for missing hands
- A sequence classifier with accuracy reported on signers held out of training entirely
- A live app running at a usable frame rate on a normal laptop, showing the recognised sign and confidence
- A confusion analysis identifying which signs are mistaken for which, and why (handshape, movement or location)
- A comparison against a baseline trained on raw frames or on hands only, to justify the landmark design
- A short design note written with input from deaf signers on what the tool should and should not claim
If you have time left
- Short phrase recognition from a few chained signs
- A guided practice mode that tells a hearing learner which part of their sign differed from reference examples
- Fully client-side browser deployment so no video leaves the device
- Transfer to a second sign language with a small dataset, measuring how much data is needed
The problem sign language users face
Deaf signers have very little software that meets them where they are. Most accessibility tooling assumes speech or text, and the gap is widest in everyday, unplanned interactions: a pharmacy counter, a classroom, a government office. Hearing family members and teachers who want to learn often have no feedback on whether their signs are right.
Sign languages are full languages, and they differ by country. Pakistan Sign Language (PSL) is not the same as American or British Sign Language, and far less software supports it.
What already exists and the gap
- PSL.org.pk, run by the Family Educational Services Foundation's Deaf Reach programme, offers a free PSL dictionary of thousands of words, stories and tutorials, as video, not recognition.
- Hand Talk translates text and speech into Brazilian and American Sign Language with a 3D avatar, which is the opposite direction to recognition.
- Research datasets such as WLASL (American Sign Language) support isolated sign recognition research.
- MediaPipe from Google provides real-time hand and body landmark tracking that runs on laptops and phones.
The gap: recognition tools for regional sign languages are rare, and most student projects report accuracy on the same people they trained on. A small, honest system for isolated signs in one regional sign language, tested on unseen signers and designed with deaf users, is valuable and achievable.
What you build
Users: deaf signers who sign to the camera, and hearing learners who practise.
- Data collection tool that records signs with prompts and consent.
- Landmark extraction and normalisation (translate, scale, handle missing hands).
- Sequence classifier over landmark sequences.
- Live app showing recognised signs as text.
Why landmarks rather than raw video
Working from keypoints rather than pixels reduces the data needed enormously, trains on hardware you actually have, generalises better across lighting and clothing, and protects privacy because you can store landmarks rather than faces. It is the difference between a project that works and one that runs out of time.
The technically interesting part
The hard problem is signer independence. A model trained on three people scores well on those three and fails on a fourth. This is the single most common flaw in student projects on this topic. Hold at least two signers out of training entirely and report that number as your headline result. Normalisation of body size and position, augmentation such as small rotations and speed changes, and more signers matter more than a bigger model.
The second is segmentation in a live stream: deciding when a sign starts and ends from continuous frames, so the app does not output a sign every few frames.
How to evaluate it
- Held-out signer accuracy (top-1 and top-5) as the headline number, next to seen-signer accuracy.
- Confusion matrix and a written explanation of the top confusions.
- Ablation: landmarks versus hands only versus raw frames.
- Live performance: frames per second and latency from sign end to text on a named laptop.
- User session: deaf signers try the live app and rate whether it recognises their signing; record their feedback on what to change.
Suggested plan
- Weeks 1 to 4: contact a deaf school or association, choose the 50 signs with them, build the recording tool.
- Weeks 5 to 9: data collection across signers; landmark pipeline.
- Weeks 10 to 14: first classifier, held-out signer evaluation.
- Semester 2, weeks 1 to 5: normalisation, augmentation, more signers, ablations.
- Weeks 6 to 10: live app with segmentation, browser or laptop deployment.
- Weeks 11 to 14: user sessions, confusion analysis, report.
Path to a product
The most realistic first use is education: a practice tool for hearing parents, teachers and staff learning a regional sign language, licensed to deaf schools, NGOs and organisations that train customer-facing staff. The recorded, consented dataset itself has research value if released with signers' permission. Be cautious about claiming translation; it sets expectations the technology cannot meet.
Scope warning
- Continuous signing, with grammar, facial grammar markers and coarticulation, is an open research problem. Say in your first sentence that you do isolated signs.
- Do not train and test on the same signers.
- Do not build a two-way translation app or an avatar.
- Involve deaf signers in the design; projects built entirely by hearing students often solve a problem the community does not have.
Learn what you need on Sefism
- Machine learning roadmap : sequence models, splits and evaluation.
- Machine learning cheat sheet : confusion matrices and metrics at a glance.
- Google Colab : free GPUs for training sequence models.
- Learn Accessibility : accessibility principles for the app itself.
- Google Research Blog : research writing on on-device perception and accessibility.
Get help with this idea
The quality of this project rests on signer independence and on working with deaf signers from the start. Tauseef can help you plan data collection, the held-out evaluation and a scope your team can finish. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.