AI Agent Readiness Audit for Websites: Can a Browser Agent Complete Your Key Tasks?

The problem: sites that people can use but agents cannot

Browser agents from the major AI companies can now open websites, click, type and complete tasks for users. A shopper may soon ask an agent to "book the earliest dental appointment near me" instead of visiting the clinic site. If the agent cannot find the booking button because it is an unlabelled icon, or gets stuck in a date picker that only works with a mouse drag, the business loses that customer without ever knowing why.

Small businesses, clinics, schools and local shops in Pakistan and elsewhere have no way to test this. They do not know which of their pages agents fail on, or what to change. The same pages are often hard for screen reader users too, because many agents and assistive technologies both read the accessibility tree.

What already exists and the gap

• Agent benchmarks such as WebArena and its successors measure how good agents are on fixed test sites. They grade the agent, not the website.
• Accessibility checkers (axe-core, Lighthouse, WAVE) find rule violations, many of which also confuse agents, but they do not show whether a real task can be completed.
• WebMCP, a proposal from the Chrome and Edge teams in a W3C community group, lets sites expose tools to browser agents directly. It is new, and most sites will not adopt it for years.
• End-to-end testing tools such as Playwright run scripted flows, not open-ended agents.

The gap is a tool that flips the benchmark around: fix the agent, vary the website, and tell the owner exactly which parts of their page cause failures.

Inspiration

WebArena (Zhou et al., ICLR 2024) built realistic self-hosted websites and 812 tasks with programmatic checks of whether each task succeeded. Borrow its method: tasks written in plain language, a success check that does not rely on the agent's own claim, and repeated runs. Then apply it to the website side of the problem.

What you build

• Site owners or their developers write five to ten key tasks in a simple file, with how success is checked (an order exists, a confirmation page appears, a form submission reaches a test inbox).
• The runner sends an agent to attempt each task several times and records everything.
• The report shows a readiness score, the steps where agents failed most, and the page cause behind each failure, with a fix such as "add an accessible name to this button" or "replace the drag-only date picker".

The technically interesting part

• Attributing failure to the page. An agent can fail because the model is weak or because the page is hostile. Run the same task with a page fix applied (for example, injecting an accessible name) and measure whether success rises. That counterfactual test is what makes the report trustworthy.
• Variance. Agents are not deterministic. Report success rates with confidence intervals, not single runs.
• Cost control. Cap steps and tokens per task, cache page snapshots, and use a small model for classification.
• Safety. Never let the agent submit real payments or messages. Run against staging copies, or intercept final submissions.

How to evaluate it

• Agreement: for 100 failed steps, compare your classifier's cause with a human label.
• Fix impact: success rate on each task before and after the suggested fixes are applied.
• Stability: score variance across repeated runs of the same site.
• Overlap: how many agent failures were also flagged by an accessibility checker, and how many were new.

Suggested plan

• Weeks 1 to 4: read WebArena, set up a browser agent, clone three small sites locally.
• Weeks 5 to 9: task format, runner, traces and success checks.
• Weeks 10 to 14: failure classifier and the counterfactual fix tests.
• Weeks 15 to 20: dashboard, CI mode, the 15-site study.
• Weeks 21 to 26: analysis, write-up and demo.

Path to a product

Web agencies already sell accessibility and SEO audits. An "agent readiness" audit is a natural next service as agent traffic grows, and a hosted runner with monthly re-checks is a clear subscription. The study is also publishable, and few student projects show this mix of agents, browsers and evaluation.

Scope warning

Do not build your own agent framework; use an existing one and keep it fixed. Do not test against live production checkouts or payment pages. Keep the task set small and the success checks strict, because vague checks make every number meaningless. Five well-chosen tasks on fifteen sites, each run ten times, is already a substantial and honest study.

Learn what you need on Sefism

• AI agents and MCP roadmap: how agents plan, act and fail.
• AI agents and MCP cheat sheet: tools, traces and evaluation in one place.
• How do I prove my AI agent project actually works?: the evaluation mindset this project depends on.
• Playwright Documentation: driving the browser and capturing traces.
• How to Meet WCAG 2.2: many fixes in your report will map to these rules.
• OWASP Top 10 for Agentic Applications: keeping the agent from doing harm while it tests.

Get help with this idea

The risk is a tool that measures the agent instead of the website, or numbers that change every run. Tauseef can help you design the counterfactual tests, choose tasks and keep the study honest. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session