Fair Scraper Gate for Small Websites with Adaptive Proof of Work and Signed AI Agent Verification

The problem: scrapers that take a small site down

Since 2024, many small websites have been hit by crawlers collecting training data for AI models. They ignore robots.txt, rotate through thousands of IP addresses and request every page, every commit and every search result. Open source code forges, university course pages, digital archives and community wikis have reported slowdowns and outages caused by this traffic. The people running them are often volunteers or a single IT officer.

Blocking everything automated is not the answer either. Search engines, accessibility tools, research crawlers and the new generation of AI agents acting for real users all have legitimate reasons to visit. And the people most hurt by heavy browser challenges are users on cheap phones and slow connections, which describes much of the audience in Pakistan and South Asia.

What already exists and the gap

• Anubis, an open source tool released in January 2025 and now maintained by Techaro, places a SHA-256 proof-of-work challenge in front of a site. It has been adopted by projects such as GNOME's GitLab, FFmpeg and the Linux kernel mailing list archives.
• Cloudflare offers bot management, Turnstile challenges and support for Web Bot Auth, an IETF draft built on HTTP Message Signatures (RFC 9421) that lets bots and agents prove who they are with a signature and a published key.
• Rate limiting in NGINX or a web application firewall helps against single sources, but not against traffic spread over thousands of addresses.

The gap your project explores is a self-hosted gate that combines both ideas (make anonymous bulk crawling expensive, let identified agents through under clear rules) and treats fairness to low-end devices as a measured requirement, not an afterthought.

Inspiration

Two sources shape this project. Adam Back's Hashcash (2002) proposed proof of work as a cost on senders of bulk email: cheap for one message, expensive for millions. Anubis applies the same economics to web requests. Web Bot Auth, documented by Cloudflare and proposed at the IETF, shows how a well-behaved agent can sign its requests so a site can verify it without guessing from the user agent string. Your project brings the two together and measures the trade-offs.

What you build

• Site operators run the gate as a container in front of their site, choose a preset, and watch dashboards.
• Visitors usually see nothing; under suspicious load, they see a short, accessible challenge that solves in the background.
• Agent operators sign their requests, register their key directory, and get predictable access under published rate limits.

The technically interesting part

• Adaptive difficulty. A fixed challenge either annoys everyone or stops nobody. Raise difficulty for expensive paths (search, archives, diffs) and for clients that request many pages quickly, and lower it for returning visitors with a valid token.
• Fair to cheap phones. Measure solve time on real low-end phones. Explore memory-hard puzzles, a cap on worst-case solve time, and a no-JavaScript path such as a simple wait page.
• Signature verification. Fetch and cache key directories, check signature windows and replay protection, and handle key rotation.
• Low overhead. The proxy must add very little latency for normal traffic. Benchmark it.
• Honest metrics. Separate "challenged", "solved", "verified agent" and "blocked", and estimate how much origin load was saved.

How to evaluate it

• Origin protection: origin CPU and response times during a replayed scraper load, with and without the gate.
• Human cost: median and worst-case solve times and battery use on three phone classes.
• Agent path: correctness of signature verification against valid, expired, replayed and forged requests.
• Overhead: added latency at the 50th and 99th percentile for normal traffic.

Suggested plan

• Weeks 1 to 4: read Hashcash, the Anubis documentation and the Web Bot Auth drafts; build a basic proxy.
• Weeks 5 to 10: challenge page, token issuing and adaptive difficulty.
• Weeks 11 to 15: signature verification, policy rules and metrics.
• Weeks 16 to 20: load tests, low-end phone study and tuning.
• Weeks 21 to 26: deployment guide, write-up and demo.

Path to a product

Universities, public archives, NGOs and small hosting companies all need protection they can run themselves. A free open source gate with paid support, or a managed version for hosting providers, is a realistic path. The fairness study is useful to the wider community, and the project is a strong portfolio piece for backend, security and infrastructure roles.

Scope warning

Do not try to build a full bot management platform or fingerprint users; that raises privacy problems and is a different project. Keep the policy language small. Test only against your own sites and replayed traffic, never against third-party services. Fairness on low-end devices is the core of this project, so measure it from the start.

Learn what you need on Sefism

• Backend development roadmap: proxies, HTTP and caching.
• Cybersecurity roadmap: signatures, replay protection and abuse handling.
• MDN: HTTP: headers, cookies and status codes the gate relies on.
• Go by Example: if you write the proxy in Go.
• Grafana k6 Documentation: replaying scraper-like load.
• Platform engineering and SRE roadmap: metrics, dashboards and running a service others depend on.

Get help with this idea

The risk is a gate that punishes real users more than scrapers, or a study with no real measurements. Tauseef can help you set the fairness targets, design the load tests and scope the policy rules. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session