On-Device Speech Recognition for an Under-Served Language
An offline speech recognition app for speakers of a language or dialect that current models handle poorly, such as Punjabi, Sindhi, Pashto or accented Urdu, built by fine-tuning an open model and running it on a mid-range Android phone. A deep learning final year project where the data work matters as much as the model.
Advanced
Needs a team, real planning, and a supervisor who knows the area.2 semesters · 2 to 3 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- A labelled speech dataset of at least 10 hours, assembled from open sources and consented recordings, with a documented collection and transcription guide
- A test set whose speakers never appear in training, balanced by gender, age group and recording condition
- Word and character error rate of the base model and your fine-tuned model on that test set
- A quantised model running offline on a named mid-range Android phone, with measured latency, memory use and model size
- A recording and transcription app that works with no network connection
- An error analysis by speaker, accent, recording condition and code-switching with English
- A data statement covering licences, consent and what the dataset should not be used for
If you have time left
- Speaker diarisation for two-speaker recordings
- Punctuation and casing restoration as a second stage
- A correction interface whose edits, with consent, feed back into the training set
- A comparison of Whisper against MMS as the base model on the same data
The problem speakers of under-served languages face
Speech recognition works well for a small number of well-resourced languages and noticeably worse for everything else, including most regional languages and dialects. A farmer who speaks Saraiki, a shopkeeper who switches between Punjabi and Urdu mid-sentence, or an elderly person who cannot type all get poor results, if they get any. They are often the people who would benefit most from voice input.
Connectivity makes it worse. Cloud transcription needs data, and data is unreliable exactly where these languages are spoken. The gap is largely a data problem rather than an algorithmic one, which makes it tractable for a student team.
What already exists and the gap
- OpenAI Whisper is an open multilingual speech model, strong on high-resource languages and uneven on low-resource ones.
- Meta's Massively Multilingual Speech (MMS) covers speech recognition for over a thousand languages, with quality that varies widely.
- Google speech services support many South Asian languages in the cloud.
- Mozilla Common Voice is an open, crowd-sourced voice dataset with recordings in many languages, including Urdu.
The gap is a model that is fine-tuned and measured for one specific under-served language or dialect, evaluated on speakers it has not heard, and small enough to run on a phone with no connection. Honest error analysis by speaker and condition is itself a contribution.
What you build
Users: speakers of the target language dictating notes or messages, and you as the data team.
- Data pipeline: collect, clean, segment and transcribe audio with a consistent guide.
- Fine-tuning pipeline on an existing open model.
- Quantised on-device model and a simple Android app.
The technically interesting part
Two hard problems.
The data. Ten hours of clean, consistently transcribed speech is more work than anyone expects. Spelling conventions, code-switching and dialect words need a written transcription guide, or your error rate will measure inconsistency rather than model quality. Plan collection in the first three weeks and get consent from everyone you record.
On-device trade-offs. Getting a model small and fast enough for a mid-range phone involves real trade-offs between accuracy, size and latency. Measuring those trade-offs across model sizes and quantisation levels turns a notebook into engineering.
How to evaluate it
- Word and character error rate against the base model on a test set of speakers never seen in training. Character error rate matters for languages with inconsistent spelling.
- Breakdown by speaker, gender, recording condition (quiet, street, phone call) and code-switched utterances. An average hides the cases that matter.
- On-device table: error rate, model size, real-time factor and peak memory per quantisation level.
- User test: five native speakers dictate short messages and rate how many corrections they needed.
Suggested plan
- Weeks 1 to 3: choose the language with native-speaker team members; write the transcription guide; gather open data.
- Weeks 4 to 9: record and transcribe consented speech; build the held-out test set.
- Weeks 10 to 14: baseline evaluation of Whisper and MMS; first fine-tune.
- Semester 2, weeks 1 to 5: improve with more data and augmentation; error analysis.
- Weeks 6 to 10: quantisation and the Android app.
- Weeks 11 to 14: user test, data statement, report.
Path to a product
The first customers are organisations that talk to speakers of the language every day: NGOs running field surveys, health programmes whose workers write visit notes, and call centres. Offer an offline dictation SDK or app licensed per organisation. Releasing the test set and model openly, with contributors' consent, builds credibility and invites the community to improve it.
Scope warning
- Do not train from scratch. Fine-tuning an existing open model is the correct approach.
- One language or dialect.
- Do not scrape audio without clear licences, and do not record anyone without consent.
- Do not build translation or text-to-speech.
Learn what you need on Sefism
- Machine learning roadmap : fine-tuning, evaluation and error analysis.
- Mobile roadmap : the Android side of running a model on the device.
- Google Colab : free GPUs for fine-tuning speech models.
- Android Developers: Develop for Android : audio capture and performance on real devices.
- Guide to App Architecture : structure the app so the model runs off the main thread.
Get help with this idea
This project succeeds or fails on the transcription guide, the held-out test set and consent. Tauseef can help you choose a language your team can handle, plan data collection and scope the on-device work. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.