Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper
by Crawl4AI
Turn any website into clean, LLM-ready Markdown for RAG pipelines and agents, without renting a scraping API.
Overview
Crawl4AI's documentation (v0.9.x at the time of writing) is organised into five parts. Setup & Installation covers installation and a self-hosting guide. Core is the bulk of the reading: command line interface, simple crawling, deep crawling, adaptive crawling, URL seeding, domain mapping, the declarative C4A-Script, the CrawlResult object, browser/crawler/LLM configuration, Markdown generation and fit-Markdown, page interaction, content selection, cache modes, local files and raw HTML, and link and media handling. Advanced adds adaptive strategies, virtual scroll, file downloading, lazy loading, hooks and auth, proxy and security, anti-bot and fallback, undetected browser, session management, multi-URL crawling, the crawl dispatcher, identity-based crawling, SSL certificates, network and console capture, and PDF parsing. Extraction documents LLM-free strategies (CSS/XPath schemas, BM25 filtering, cosine similarity clustering) alongside LLM strategies and chunking, so token spend is a choice rather than a default. The API reference covers AsyncWebCrawler, arun(), arun_many(), the config objects, CrawlResult, strategies and C4A-Script. The library is built on Playwright and supports Chromium, Firefox and WebKit, and ships a Docker image with a FastAPI server and JWT auth for deployment. The project is Apache-2.0 with roughly 81.8k GitHub stars and about 1,651 commits on main; the most recent release, v0.9.3, was a security release closing five vulnerabilities in PDF processing and the Docker Playground plus 33 bug fixes, with earlier releases adding deep-crawl crash recovery, prefetch URL discovery, and proxy-escalating anti-bot detection.
At a Glance
- Topic
- RAG
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~5-7 hours to work through Core, Advanced and Extraction; self-paced reference thereafter
- Provider
- Crawl4AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Generate clean, citation-annotated Markdown from arbitrary pages for RAG ingestion
- ✓Run LLM-free extraction with CSS and XPath schemas to avoid per-page model costs
- ✓Configure deep crawling and adaptive crawling that stops once information gain flattens
- ✓Handle lazy loading, virtual scroll and page interaction before content is captured
- ✓Survive bot defenses using undetected browser mode, proxy rotation and anti-bot fallback
- ✓Scale concurrent jobs with arun_many, the crawl dispatcher and session management
- ✓Deploy the crawler as a self-hosted Docker and FastAPI service with JWT authentication
- ✓Chunk and cluster extracted content with BM25 filtering ahead of embedding
Highlights
- •Apache-2.0 with ~81.8k GitHub stars — no API key and no per-page pricing, unlike hosted scraping services
- •Adaptive crawling uses information-foraging heuristics to stop when a query is answered, instead of crawling to a fixed depth
- •Ships both LLM-free (CSS/XPath/BM25) and LLM-driven extraction, so you spend tokens only where structure genuinely needs a model
- •v0.9.3 was a security release closing five PDF-processing and Docker Playground vulnerabilities plus 33 bug fixes — the project patches rather than accretes
- •C4A-Script gives a declarative way to script page interaction without hand-writing Playwright
Who It’s For
Best For
- ✓AI engineers building a self-hosted document ingestion layer for RAG
- ✓Teams replacing a paid scraping API to cut per-page crawl costs
- ✓Agent developers who need a browse-and-fetch tool that returns Markdown
- ✓Data engineers assembling training or evaluation corpora from the open web
Prerequisites
- •Comfortable with async Python (asyncio, async/await)
- •Basic familiarity with CSS selectors or XPath for extraction schemas
- •Playwright installs a browser runtime, and the deployment path assumes some Docker knowledge
FAQ
What is Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper?
The official documentation for Crawl4AI, the Apache-2.0 async Python crawler that converts web pages into clean, citation-annotated Markdown and structured JSON for RAG pipelines and AI agents. It is written for engineers who need to own their ingestion layer rather than rent one, and it runs from a five-line first crawl through session management, anti-bot fallbacks and Docker deployment. After it you can stand up a repeatable self-hosted crawl-to-corpus pipeline.
Is Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper free?
Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper is free to access.
What level is Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper for?
Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper is aimed at a intermediate audience. Recommended background: Comfortable with async Python (asyncio, async/await), Basic familiarity with CSS selectors or XPath for extraction schemas, Playwright installs a browser runtime, and the deployment path assumes some Docker knowledge.
How long does Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper take?
Expect roughly ~5-7 hours to work through Core, Advanced and Extraction; self-paced reference thereafter. Most learners work through it at their own pace.
What will I learn from Crawl4AI Documentation — Open-Source LLM-Friendly Web Crawler and Scraper?
You'll learn: Generate clean, citation-annotated Markdown from arbitrary pages for RAG ingestion; Run LLM-free extraction with CSS and XPath schemas to avoid per-page model costs; Configure deep crawling and adaptive crawling that stops once information gain flattens; Handle lazy loading, virtual scroll and page interaction before content is captured; Survive bot defenses using undetected browser mode, proxy rotation and anti-bot fallback; Scale concurrent jobs with arun_many, the crawl dispatcher and session management; Deploy the crawler as a self-hosted Docker and FastAPI service with JWT authentication; Chunk and cluster extracted content with BM25 filtering ahead of embedding.
Topics
Sources
This page was written from 3 sources, 2 on domains other than docs.crawl4ai.com.