Urdu and English Phone Booking Voice Agent for Clinics and Salons

The problem small clinics and salons have with the phone

Small clinics, dental practices, salons and tutoring centres still take most bookings by phone. The receptionist is also checking patients in, handling payments and answering questions, so calls go unanswered at peak hours and after closing. Every missed call is often a missed booking. In Pakistan and much of South Asia those calls happen in Urdu, English and a fluid mix of both, with dates like "agle mangal ko shaam chaar baje" that general voice systems struggle with.

Voice agents are now good enough to handle simple, well-defined calls. Booking an appointment is exactly that kind of call, if the agent understands dates and names correctly and never writes a wrong booking.

What already exists and the gap

• Retell AI and Vapi are platforms for building phone voice agents; public comparisons in 2026 put their typical response latency around 500 to 700 milliseconds and pricing at a few to tens of cents per minute depending on the models chosen.
• Pipecat and LiveKit Agents are open-source frameworks for real-time voice pipelines that you can self-host.
• Open speech models such as Whisper and Meta's MMS cover Urdu, with quality that varies a lot on names, numbers and code-switched speech.

The gap is not the pipeline; it is domain accuracy in code-switched Urdu and English for the specific entities a booking needs, a strict confirmation flow, and a cost that a small clinic can pay per month.

What you build

A voice agent answering a number, backed by a booking system. The caller speaks; voice activity detection segments speech; recognition produces text; a language model with tools decides what to do; text-to-speech replies. The booking backend holds services, staff calendars and slots. The agent can list available slots, book, reschedule and cancel, and must read back every change and get a clear confirmation before writing it. A handoff tool forwards the call or takes a callback request when the agent is unsure.

You spend real effort on speech recognition for this domain: collect recordings of volunteers saying dates, times, phone numbers and names in both languages, measure word and entity error rates, then fine-tune or apply prompt biasing and a post-processing step that normalises spoken dates into calendar values.

The technically interesting part

Two problems make this advanced.

Entity accuracy in code-switched speech. A general word error rate hides what matters. Define an entity error rate for dates, times, names and phone numbers, and report it for the base model, after fine-tuning, and after your date normaliser. Show which errors remain and why.

Latency under a strict budget. People hang up on silence. Stream every stage, start speaking as soon as the first sentence is ready, and handle barge-in when the caller interrupts. Present a per-stage latency breakdown and the 50th and 90th percentile end-to-end response times, the same way a production team would report a service level target.

How to evaluate it

• Task success on at least 60 scripted calls: was the correct booking created, changed or cancelled, with no wrong writes?
• Entity error rate for dates, times, names and numbers before and after adaptation.
• Median and 90th percentile response latency, with the per-stage breakdown.
• Handoff rate and the share of handoffs that were justified.
• Cost per call minute for hosted and self-hosted variants.
• A pilot or realistic trial with at least 15 people calling to book, with a short satisfaction survey afterwards.

Suggested plan

1. Month 1: pipeline skeleton with Pipecat or LiveKit, booking backend, English-only happy path.
2. Month 2: Urdu and code-switched recognition baseline, recording collection.
3. Month 3: fine-tuning or biasing, date and number normaliser, entity error measurement.
4. Month 4: confirmation flow, handoff, barge-in and latency work.
5. Month 5: scripted call harness and adversarial calls.
6. Month 6: pilot with real callers, cost analysis, write-up.

Path to a product

The first customer is a single-doctor clinic or a salon that misses calls at peak time; a free month of after-hours answering is an easy pilot. Charge a flat monthly fee with a minutes allowance, bundled with the booking calendar. Scaling means more languages, more verticals with their own booking rules, and integrations with existing clinic software.

Scope warning

Do not give medical advice, triage symptoms or discuss prescriptions; the agent books appointments only and hands off everything else. Do not clone anyone's voice. Do not try to build your own speech models from scratch; adapt open ones. Record calls only with clear consent announced at the start.

Learn what you need on Sefism

• AI Engineering roadmap : LLM tool calling, evaluation and latency trade-offs.
• Machine Learning roadmap : fine-tuning speech models and measuring error rates.
• System Design roadmap : streaming pipelines and latency budgets.
• roadmap.sh: AI Agents : agent concepts, tools and handoff patterns.
• LiteLLM Documentation : switch and cost different LLM providers behind one interface.

Get help with this idea

Voice agents are easy to demo and hard to make reliable. A mentor can help you define the scripted call suite and latency targets so your evaluation is convincing. Bring your draft scope, the dataset you plan to use and your evaluation table, and we can look at them together. The build is yours; the guidance is on scope, stack and how to measure it. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session