Crawl4AI

What Crawl4AI is

Crawl4AI is an open-source web crawler and scraper built for feeding websites to language models. Its README sums it up: any website into clean, LLM-ready Markdown. You run it yourself, as a Python library or as a Docker service, which keeps costs and data under your control.

Key features

• Clean Markdown: headings, lists, tables and code blocks preserved, with filters that strip navigation and boilerplate, and links turned into a numbered reference list.
• Extraction without an LLM: pull structured fields with CSS or XPath rules, which is fast, free and repeatable.
• Extraction with an LLM: when pages vary too much for rules, send the content to a model of your choice with a schema.
• A real browser: Chromium, Firefox or WebKit, JavaScript execution, scrolling, persistent profiles, proxies and custom headers.
• Crawling strategies: breadth-first, depth-first and best-first crawls, adaptive crawling that stops when it has enough, crash recovery and batch processing.
• Docker deployment: a REST API for Markdown, HTML, screenshots and PDFs, a monitoring dashboard, and MCP integration so agents can call it.

Getting started

Install with pip install -U crawl4ai, then run crawl4ai-setup to install the browser it needs. A few lines of async Python fetch a page as Markdown.

When to choose it

Pick Crawl4AI when you want a free, self-hosted crawler, need fine control over the browser, or must keep crawled data on your own servers. Pick a hosted API when you would rather not run browsers yourself.

Use it responsibly

Check each site's terms and robots rules, crawl slowly, prefer official APIs or exports where they exist, and do not collect personal data you have no right to.

Prerequisites and time

Comfortable Python, including async functions, and a little HTML and CSS for selector-based extraction. A first crawl takes an afternoon.

Free or paid?

Free and open source under the Apache 2.0 licence.

Common questions

Do I need an LLM to extract data? Often not. For pages with a stable layout, CSS or XPath rules are faster, free and give the same result every time. Use an LLM only when pages vary too much for rules.

Best for: students and teams building RAG or research tools over web content who want an open-source, self-hosted crawler.