On-Device LLM Benchmark App for Budget Android Phones with Urdu Quality Tests
Build an Android app that runs small language models on the phone itself and measures what developers need to know before shipping on-device AI: speed, memory, battery drain, heating over a sustained run, crashes, and answer quality on Urdu and Roman Urdu tasks. Results from volunteers' phones feed a public table of how common budget models actually perform. Most published numbers come from flagship phones and English prompts, which tells a developer in Pakistan very little.
Advanced
Needs a team, real planning, and a supervisor who knows the area.2 semesters · 2 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- An Android app that downloads a chosen small model in a quantised format and runs a fixed benchmark suite
- Measurements of time to first token, prompt processing speed, generation speed and peak memory
- A sustained ten-minute run that records speed, battery level and device temperature to show throttling
- An Urdu and Roman Urdu task set, such as short summaries, classification and field extraction, scored automatically on the phone
- Crash and out-of-memory detection with a clear report instead of a silent failure
- Anonymous result upload with explicit consent and a public results table by phone model and model size
- A report comparing at least five real budget and mid-range phones and two runtimes
If you have time left
- An iOS version for comparison
- Testing the phone's NPU or GPU path where the runtime supports it
- A recommendation screen: which model size this phone can run comfortably
- Tracking how results change across runtime versions
The problem: will it run on my users' phones?
On-device AI is attractive for apps in Pakistan: no data leaves the phone, it works without mobile data and there is no per-request API bill. Developers building a farm diary, a health app or a shop ledger ask the same questions. Will a small model run on the phones our users actually own, with three or four gigabytes of memory? How fast? Will it drain the battery or heat the phone until it slows down? And can it handle Urdu at all?
Published benchmarks rarely answer this. They tend to use flagship phones, short runs and English prompts.
What already exists and the gap
- Google AI Edge Gallery is an open source app for running open models on Android phones offline, with benchmark views.
- PocketPal AI runs small models from several providers on Android and iOS.
- Research benchmarks have measured on-device language models across phones and frameworks.
These show that models can run. The gap is a benchmark focused on budget phones common in South Asia, sustained use rather than a single prompt, Urdu and Roman Urdu quality, and a shared table of results from real devices that developers can consult before choosing a model.
Inspiration
"MELTing Point: Mobile Evaluation of Language Transformers" by Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto and Hamed Haddadi, presented at ACM MobiCom 2024, was a systematic study of running language models on phones and edge devices, measuring performance, energy and accuracy across models and frameworks, with open code. It found that on-device execution affects latency, device stability and energy use. Your project applies a similar method to the phones and languages that matter locally.
What you build
- App developers look up a phone model and see which model sizes run well, with speed, memory and heat numbers.
- Volunteers install the app, run the suite overnight on charge or during a set test, and choose whether to share results.
- Researchers get an open dataset of on-device results from budget phones.
Every result records the phone model, Android version, runtime version, model file and quantisation, and the test conditions. Without these, numbers cannot be compared. The public table lets a developer filter by memory size or price band and sort by sustained speed, and each row links to the raw traces so anyone can check how it was measured.
The technically interesting part
- Measuring fairly. Phones vary in background apps, battery state and room temperature. Define a test protocol: charge level, airplane mode, screen brightness, cool-down before each run. Record conditions with every result.
- Thermal throttling. A model may generate quickly for the first minute, then slow sharply as the phone heats. A sustained run with a speed and temperature trace is more useful than a single number.
- Memory limits. Android may kill an app that uses too much memory, sometimes without a clear error. Detect it and record it as a result, not a crash.
- Urdu quality. Build a small task set with gold answers: summarising a short Urdu message, extracting an amount and date, classifying a Roman Urdu complaint. Score answers automatically on the phone and check a sample by hand.
How to evaluate it
- Repeatability: the same phone, model and protocol run five times should give similar numbers; report the spread.
- Coverage: results from at least five distinct phone models, including two budget phones.
- Quality: Urdu and Roman Urdu task scores per model, compared with a hosted model as a reference.
- Usefulness: ask two or three app developers to use the results table to choose a model, and record what they decided and why.
Suggested plan
- Weeks 1 to 4: read MELTing Point, build a minimal Android app running one model with llama.cpp.
- Weeks 5 to 9: measurements, the test protocol and the sustained run.
- Weeks 10 to 14: Urdu task set and on-phone scoring.
- Second semester, weeks 1 to 5: second runtime, crash handling, results upload.
- Weeks 6 to 10: collect results from volunteers' phones.
- Weeks 11 to 14: analysis, public table, report and demo.
Path to a product
Every company considering on-device AI for emerging markets faces this choice. A maintained public benchmark with paid custom testing on a client's own model and target phones is a realistic service. The project also builds deep Android performance skills, which are scarce and valued.
Scope warning
Do not train or fine-tune models; benchmark existing ones. Keep the Urdu task set small and well checked rather than large and noisy. Collect results only with clear consent, never collect personal data, and make sure the app cannot run long tests on battery without the user knowing.
Learn what you need on Sefism
- Mobile development roadmap: Android app structure and performance basics.
- Mobile development cheat sheet: quick reference for lifecycle, storage and background work.
- AI engineering roadmap: model choice, quantisation and evaluation.
- Android Developers: Develop for Android: memory, battery and thermal APIs.
- Kotlin Documentation: the language the app is written in.
- Ollama: trying the same models on a laptop before moving them to the phone.
Get help with this idea
The risk is numbers nobody can trust because the test conditions varied. Tauseef can help you design the protocol, choose models and phones and present the results clearly. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.