Cloned Voice Detection for Urdu Phone Calls and Voice Notes

The problem: a familiar voice asking for money

A mother gets a WhatsApp voice note from her son: he has been in an accident and needs money sent to a wallet number now. It sounds exactly like him. A shopkeeper gets a call from "the bank" in a voice that sounds like the branch manager. Voice cloning tools can now copy a voice from a short sample, and the samples are easy to find on social media and in voice notes forwarded around family groups.

People in Pakistan already face a steady stream of wallet and banking scams. Cloned voices make them far more convincing. Banks that use voice in their call centres face the same problem from the other side.

What already exists and the gap

• Anti-spoofing research has run public challenges for years, with large datasets of real, synthesised and converted speech, mostly in English.
• Commercial call-centre fraud detection products analyse calls for signs of synthetic voices, sold to large banks and telecom companies.
• Open source cloning tools improve every few months, and each new one can fool detectors trained on older tools.

The gap is honest evidence for Urdu: how well do detectors trained on English studio audio work on Urdu speech that has passed through a phone line or a messaging app, and how much does training on a small Urdu set help, especially against cloning tools the detector has never seen?

Inspiration

"ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech" by Xin Wang, Junichi Yamagishi, Massimiliano Todisco and many co-authors, published in Computer Speech and Language in 2020, describes the dataset and protocol behind one of the main anti-spoofing challenges, including an evaluation where some attack methods are unseen in training. Your project follows the same protocol thinking for Urdu and adds the phone and messaging channel that real scams use.

What you build

• Banks and telecom fraud teams get a tested Urdu detector and, more importantly, numbers on where it fails.
• Researchers get a consented Urdu dataset and evaluation protocol, released under clear terms if consent allows.
• The public, through a simple page, can check a suspicious voice note and see a cautious score with the advice to call the person back on a known number.

The page must never say "this is definitely real". Detectors make mistakes, and a false "real" verdict could help a scammer. The advice to verify through another channel is always shown, whatever the score.

The technically interesting part

• The channel destroys evidence. Many detectors rely on fine detail in high frequencies that phone codecs and voice note compression remove. Simulate the channels properly and measure the drop.
• Unseen tools. A detector that learns the fingerprint of one cloning tool fails on the next. Leave-one-tool-out tests show whether your detector learned something general.
• Pretrained encoders. Self-supervised speech models trained on many languages give strong features for small datasets. Compare a few and fine-tune carefully to avoid overfitting to your speakers.
• Speaker leakage. If the same speaker appears in training and test sets, results look better than they are. Split by speaker.

How to evaluate it

• Equal error rate on clean audio, phone-channel audio and voice note audio, for the baseline and your detector.
• Unseen tool results from the leave-one-tool-out tests.
• Short clip performance at different clip lengths.
• Calibration: whether a score of 0.9 really means about nine in ten clips are clones.

Suggested plan

• Weeks 1 to 4: read the ASVspoof paper, run a published baseline, write the consent form and get ethics approval.
• Weeks 5 to 10: record volunteers, generate clones, build the channel simulation.
• Weeks 11 to 14: baseline evaluation on Urdu data.
• Second semester, weeks 1 to 6: fine-tuning, leave-one-tool-out tests, calibration.
• Weeks 7 to 10: API and the public check page.
• Weeks 11 to 14: report, data statement and demo.

Path to a product

Banks, wallets and telecom operators in Pakistan need protection against voice fraud in Urdu and regional languages, and vendors focused on English will be slow to serve them. A detection API tested on local channels, or a consulting evaluation service for banks' existing tools, is realistic. The work is excellent preparation for AI security roles.

Scope warning

Only clone voices of volunteers who have signed consent for exactly this use, and never release cloned audio or a cloning pipeline. Do not record real calls. Do not promise that the detector stops scams; present it as one signal alongside verification habits. Keep to Urdu until the protocol is solid.

Learn what you need on Sefism

• Machine learning roadmap: evaluation, data splits and calibration.
• Cybersecurity roadmap: fraud, threat models and responsible disclosure.
• Speech and Language Processing, 3rd Edition: speech signal and model background.
• Hugging Face Transformers Documentation: pretrained speech encoders and fine-tuning.
• PyTorch: Learn the Basics: training the detector.
• How do I get into AI security as a student or junior engineer?: where this project can lead.

Get help with this idea

The risk is weak consent handling or a detector that only learned one tool's fingerprint. Tauseef can help you plan the ethics process, the dataset and the evaluation protocol. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session