Roman Urdu and English Customer Message Triage for Small Business Inboxes

The problem small business inboxes have

A small online store, a tuition centre or a clinic might get a few hundred messages a day across WhatsApp, Instagram and a web form. They arrive in English, in Roman Urdu, and very often in both at once: "order #4412 abhi tak nahi aya, kab tak ayega?". One or two people read everything in arrival order, so a refund dispute sits behind twenty "price?" messages and an angry customer waits a day. Nobody knows at the end of the week how many messages were about late deliveries versus wrong sizes.

The business does not need a chatbot that pretends to be a person. It needs the inbox sorted, the urgent messages on top, and the order number pulled out so the reply takes one minute instead of five.

What already exists and the gap

• Zendesk and Freshdesk offer intent detection and routing, priced per agent per month for support teams and strongest in English and major European languages.
• respond.io and similar WhatsApp inbox tools provide shared inboxes and rule-based routing on keywords, which breaks on Roman Urdu spelling variation.
• Intercom Fin resolves conversations with an LLM and charges per resolved conversation in US dollars, a price that does not fit a store with thin margins and hundreds of low-value messages a day.

Published Roman Urdu NLP work (for example the UCI Roman Urdu sentiment set with over 20,000 records, and recent QLoRA work on code-mixed offensive language) shows the language can be modelled, but there is little on customer service intents. The gap is a cheap, accurate triage layer for code-mixed messages.

What you build

A service and an inbox. Messages arrive through an API (and later a WhatsApp webhook). The service returns an intent, an urgency score, extracted entities and a confidence. The inbox groups messages by intent, applies routing rules ("payment proof goes to accounts, complaints go to the owner"), and shows extracted order numbers as clickable chips. Staff correct wrong labels in one click; corrections are stored for retraining.

The model side has two paths. A fine-tuned small encoder (XLM-RoBERTa base or mDeBERTa) is cheap to run and fast. An LLM baseline with a clear prompt and examples is easier to start with but costs money per message. A hybrid that sends only low-confidence messages to the LLM is often the sensible production answer, and measuring that trade-off is part of your result.

The technically interesting part

Robustness to spelling and script variation. "Kab tak", "kb tk" and "kab tk" mean the same thing. Show your evaluator a targeted robustness test: take the test set, apply realistic perturbations (vowel dropping, common abbreviations, switching a few words to English or to Urdu script) and report how much accuracy drops for each model. Then try one mitigation, such as training-time augmentation with the same perturbations or a transliteration normalisation step, and show the gap closing. This turns a standard classification project into one with a clear, defensible research contribution.

How to evaluate it

• Macro F1 per intent on a held-out test set, split by English, Roman Urdu and mixed messages.
• Accuracy under each perturbation type, before and after your mitigation.
• Entity extraction precision and recall for order numbers and phone numbers.
• Cost per 10,000 messages and median latency for small model, LLM and hybrid.
• A two-week pilot or a simulated shift with two staff members: time to first response on urgent messages with and without triage.

Suggested plan

1. Month 1: partner with one or two businesses, agree anonymisation, define intents, collect messages.
2. Month 2: labelling with double annotation on a subset, agreement check.
3. Months 3 to 4: fine-tune models, build LLM baseline and hybrid, run robustness tests.
4. Month 5: inbox app, routing rules, correction loop.
5. Month 6: pilot or simulated shift, retraining with corrections, write-up.

Path to a product

The first customer is a small e-commerce brand or a clinic receptionist team already drowning in WhatsApp messages. Price it as a monthly subscription per inbox with a message cap, well below per-agent helpdesk pricing. The model can also be sold as an API to helpdesk and WhatsApp inbox vendors who lack Roman Urdu support. Other code-mixed markets (Hinglish, Arabizi) follow the same recipe.

Scope warning

Do not build an autonomous bot that replies to customers, and do not build your own WhatsApp gateway; use the official platform only if a partner already has access. Do not collect real customer messages without the business's written agreement and a clear anonymisation step. Keep the intent list short; fifteen well-defined intents beat forty vague ones.

Learn what you need on Sefism

• AI Engineering roadmap : prompting, evaluation and cost control for the LLM baseline and hybrid.
• Machine Learning roadmap : fine-tuning, validation and error analysis for the classifier.
• AI Engineering cheat sheet : quick reference for model choice and evaluation patterns.
• LiteLLM Documentation : one interface to call and cost several LLM providers for your comparison.
• Google Colab : free GPU time for fine-tuning multilingual encoders.

Get help with this idea

The decisions that matter most are the intent list, the anonymisation process and how you build the robustness test. A mentor can look at these before you start labelling. Bring your draft scope, the dataset you plan to use and your evaluation table, and we can look at them together. The build is yours; the guidance is on scope, stack and how to measure it. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session