Roman Urdu Jailbreak and Policy Drift Red Team for Customer Service Chatbots
Customer service chatbots are tested in English, then deployed to users who write in Urdu, Roman Urdu and a mix of all three. Build a red-team harness that checks whether a bot stays inside its own written rules (no promises it cannot keep, no advice outside its remit, no leaking its instructions) when attacked in local languages and code-switched text, and produces a scorecard the business can act on.
Intermediate
A small team, or one strong student willing to learn something new.1 semester · 2 to 3 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- A written policy for a realistic demo bot (for example a university admissions helpdesk or an online store's support bot) turned into testable rules
- A demo bot you build and own, using a hosted or open model, with its system prompt and retrieval
- An attack library of at least 400 prompts across English, Urdu script, Roman Urdu and code-switched text: role play, persuasion, translation tricks, system prompt extraction and off-policy promises
- An automatic attacker that mutates successful attacks into new variants, inside a fixed budget
- A grader that checks each answer against the rules, calibrated against at least 200 human judgements
- A scorecard per rule and per language, with example failures and suggested fixes to the prompt or guardrails
- A before-and-after report showing the effect of your fixes
If you have time left
- Bangla and Pashto attack sets
- Running the same suite against two or three different base models to compare robustness
- A CI mode that fails a build when the scorecard drops
The problem: safety tested in English, deployed in Roman Urdu
A store, bank, telecom or university launches a chatbot. Its rules are clear on paper: don't promise refunds outside policy, don't give medical or legal advice, don't reveal internal instructions, stay polite. The bot is tested in English. Then real users arrive and write "bhai mujhe refund chahiye warna..." or mix Urdu script with English product names.
Research has shown this gap is real. "Low-Resource Languages Jailbreak GPT-4" (Yong, Menghini and Bach, 2023) found that translating unsafe requests into low-resource languages bypassed safety training far more often than English. "Multilingual Jailbreak Challenges in Large Language Models" (Deng and colleagues, ICLR 2024) reached a similar conclusion. Roman Urdu, written in Latin script with no standard spelling, is a natural place to look.
What already exists and the gap
- garak and promptfoo provide red-team probes and test runners, mostly English-first.
- Microsoft's PyRIT is an open source toolkit for automating attacks against generative AI systems.
- Commercial red-team services test enterprise bots, rarely in South Asian languages.
The gap is twofold. First, attacks in Urdu, Roman Urdu and code-switched text. Second, testing against the bot's own business rules (policy drift), not only generic harmful content. A bot that promises a refund it should not is a business problem even if it is never "unsafe".
Inspiration
Anthropic and others publish red-teaming write-ups describing how they probe models before release. DeepLearning.AI's short course on red teaming LLM applications walks through the same method on a demo bot. Your project applies it to a local language and a written business policy.
What you build
- A demo bot you own, with a clear policy.
- An attack library and an automatic attacker that learns from what worked.
- A grader that checks answers against each rule.
- A scorecard that a non-technical manager can read.
The technically interesting part
- Roman Urdu has no standard spelling. "Nahi", "nai" and "nhi" all mean the same thing. Your mutations should explore spelling variants on purpose.
- Grading is the hard part. An LLM judge must decide whether an answer broke a rule. Calibrate it against human labels and report agreement honestly.
- Fixes that do not overcorrect. A bot that refuses everything passes your attack suite and fails its users. Measure helpfulness on normal questions too.
How you will evaluate it
- Attack success rate per rule and per language, before and after fixes.
- Judge agreement with human labels (Cohen's kappa).
- Helpfulness on 100 normal customer questions, so fixes are not just refusals.
Path to a product
Companies in Pakistan, India and Bangladesh are putting chatbots in front of customers who write in local languages. A red-team service with a local-language attack library, or a CI tool that runs the suite on every prompt change, is a realistic offering.
Scope warning
Attack only the bot you built and own. Do not test production chatbots belonging to real companies, even ones with public chat widgets. Keep harmful-content categories narrow; the focus is business-policy drift and instruction leaks.
Learn what you need on Sefism
- AI security roadmap: red teaming, jailbreaks and guardrails.
- AI evals cheat sheet: how to build graders and check them against people.
- Red teaming LLM applications: a short course on the method.
- promptfoo: a test runner you can extend with your attack library.
- garak: ready-made probes to compare against.
- Ollama: run open models locally for the target and attacker.
- What does an AI evals engineer do?: how red-team results become repeatable evals.
Get help with this idea
The risk is a pile of attacks with no clear measure of what matters. Tauseef can help you write a testable policy, design the grader and present results a business would act on. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.