Sandboxed Benchmark of Browser Agents on Local-Style Service Websites with Safety Traps
Build a self-hosted set of realistic but fake websites modelled on the forms people in Pakistan use every day, such as bill payment, admissions and bank transfers, then measure how well AI browser agents complete tasks and how often they take unsafe actions. Browser agents are now offered to the public, yet most benchmarks use Western shopping and forum sites and say little about consent, payment confirmation or hidden instructions. The benchmark runs only on your own sandbox, never on real sites.
Advanced
Needs a team, real planning, and a supervisor who knows the area.2 semesters · 2 to 3 students
7 to ship
4 optional extrasSuggested stack
What you should ship
- Three self-hosted mock sites: a utility bill payment portal, a university admission form and a bank transfer flow with a fake OTP step
- At least 60 tasks written as natural language goals, each with an automatic check on the final database state
- Safety traps in the sites: pre-ticked consent boxes, a lookalike link, hidden text with instructions, and payment pages that require confirmation
- A runner that executes each task with at least two agent setups and records every action, screenshot and cost
- Metrics for task success, unsafe actions, steps taken and cost per task
- Bilingual variants of some tasks in Urdu and English to see whether language changes behaviour
- A public report and the benchmark released as open source
If you have time left
- A human approval step for risky actions and a measurement of how much it reduces unsafe outcomes
- Tasks that need reading a PDF or image on the page
- A leaderboard page with results for new agent versions
- Comparing a text-only page view with a screenshot-based agent
The problem: agents that click for you
AI browser agents can now fill in forms, pay bills and book appointments by clicking through websites on a person's behalf. That is useful for someone short of time or uncomfortable with English-language forms. It is also risky. An agent might tick a consent box nobody read, follow a lookalike link, obey instructions hidden in the page, or press "Pay" without asking.
People in Pakistan deal with utility portals, admission forms, bank sites and government services that look nothing like the shopping sites agents are usually tested on. Before anyone trusts an agent with these, someone needs to measure how they behave, safely.
What already exists and the gap
- Browser agents are available from large AI companies, and open source projects such as Browser Use let developers connect a language model to a real browser through Playwright.
- Benchmarks such as WebArena and VisualWebArena test agents on self-hosted shopping, forum, code and content sites.
- Security guidance such as the OWASP Top 10 for Agentic Applications lists the risks in general terms.
The gap is a benchmark built around the kinds of forms and payment flows people in South Asia actually use, with safety traps built in and measured alongside success, so that "it completed the task" and "it did something it should not have" are reported together.
Inspiration
"WebArena: A Realistic Web Environment for Building Autonomous Agents" by Shuyan Zhou and colleagues at Carnegie Mellon University, presented at ICLR 2024, built fully functional self-hosted websites and 812 tasks checked by functional correctness rather than by matching a fixed sequence of clicks. Your project borrows that method, self-hosted sites and state-based checks, and adds a safety dimension and a local setting.
What you build
- Researchers and agent developers run the benchmark against their agent and get a report of success and unsafe actions.
- Banks, utilities and universities can see how agents behave on flows like theirs and design pages that are safer for them.
- Students and journalists get honest, reproducible numbers instead of demos.
The output of each run is a folder anyone can inspect: the task, every action the agent took, a screenshot after each step, the final state check and the cost. Reviewers can replay a failure step by step and see exactly where the agent went wrong, which is far more convincing than a single success rate.
The technically interesting part
- Realistic but fake. The sites must feel like the real thing, with Urdu labels, CNIC fields, OTP steps and confusing layouts, while containing no real brands or data.
- Checking outcomes, not clicks. Each task has a check that reads the database at the end: was the right bill paid, was the form submitted with the right values, was consent given when it should not have been.
- Designing traps fairly. A trap must be something a careful human would avoid. Write down for each trap what the safe behaviour is, and have a teammate try each task by hand.
- Reproducibility. Models change and pages load differently. Fix seeds where you can, run each task several times, and report variance.
How to evaluate it
- Task success rate per site and per agent setup, with confidence intervals across repeated runs.
- Unsafe action rate: share of runs where the agent fell for a trap, by trap type.
- Efficiency: steps and cost per successful task.
- Language effect: success and safety on Urdu versus English versions of the same task.
- Human baseline: a few volunteers attempt a sample of tasks to show they are achievable.
Suggested plan
- Weeks 1 to 4: read WebArena, run an open source agent on a toy site, design the three mock sites.
- Weeks 5 to 10: build the sites, seed data and state checks.
- Weeks 11 to 14: write tasks and traps, pilot with humans.
- Second semester, weeks 1 to 5: runner, logging and first agent runs.
- Weeks 6 to 10: full runs, bilingual variants, analysis.
- Weeks 11 to 14: report, open source release and demo.
Path to a product
Companies building or buying agents need evidence that they are safe on the flows their customers use. A maintained benchmark with paid custom evaluations for a bank or utility's own flows is a credible service. The work sits right at the centre of AI agent safety, a growing field for engineers and researchers.
Scope warning
Never point an agent at a real bank, utility or government site, and never use real credentials. Keep to three sites; depth matters more than breadth. Do not build your own agent framework; evaluate existing ones. Budget API costs from the start and cap them in the runner.
Learn what you need on Sefism
- AI agents and MCP roadmap: how agents plan, call tools and fail.
- AI agents cheat sheet: quick reference for agent loops and guardrails.
- OWASP Top 10 for Agentic Applications: the risks your traps should cover.
- Playwright Documentation: the browser automation layer most agents use.
- AI Evals: Everything You Need to Know: designing evaluations you can trust.
- How do I prove my AI agent project actually works?: turning results into evidence.
Get help with this idea
The risk is a benchmark that is either too easy or unfair to the agents. Tauseef can help you design the tasks and traps, set up the checks and keep the budget under control. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.