Crawlberg vs doc-scraper
Crawlberg is an open-source MCP server for web search and scraping. doc-scraper is an open-source MCP server for web search and scraping.
- Crawlberg is best for feed a rag pipeline with clean site content.
- doc-scraper is best for give a coding agent the exact docs for a library version.
- What it is
- Crawlberg is an open-source MCP server for web search and scraping.doc-scraper is an open-source MCP server for web search and scraping.
- Best for
- Feed a RAG pipeline with clean site contentGive a coding agent the exact docs for a library version
- Kind
- MCP serverMCP server
- Pricing
- Open sourceOpen source
- Platforms
- CLI, self-hosted, API, Linux, macOS, WindowsCLI, macOS, Windows, Linux, self-hosted
- Runs
- LocallyLocally
- Open source
- Good for
- Feed a RAG pipeline with clean site content; Map a website's link structure and sitemap; Archive documentation as MarkdownGive a coding agent the exact docs for a library version; Keep an offline copy of documentation for air-gapped work; Detect what changed between two crawls of a docs site
What Crawlberg does
Single-page or site crawl
Scrape one URL or crawl a whole site with configurable depth, page limits and concurrency.
Markdown conversion
Pages become clean Markdown with citations and document structure.
Rich metadata
Extracts titles, links, images, JSON-LD, Open Graph, hreflang and headings.
SSRF protection
Refuses private, loopback and cloud-metadata addresses unless you opt out.
Browser fallback
Uses an optional headless browser for JavaScript-heavy single-page apps.
Many language bindings
Same core exposed to Python, Node.js, Rust, Go, Java, Ruby and others.
What doc-scraper does
Offline BM25 search
search_docs and read_page answer from the stored corpus with no network.
Framework detection
Recognises 30+ docs generators and validates the content selector.
Clean Markdown output
One Markdown file per page plus pages.jsonl, llms.txt and llms-full.txt.
Polite crawling
Honours robots.txt and sitemaps with rate limits, retries and resumable runs.
Freshness tracking
Incremental re-crawls, watch mode and crawl-to-crawl diffs.
Parallel crawls
Runs many sites at once under shared limits.
Crawlberg vs doc-scraper: questions
Is Crawlberg or doc-scraper cheaper?
Both are open source. Compare current plans on each official site.
What is the difference between Crawlberg and doc-scraper?
Crawlberg is an open-source MCP server for web search and scraping. doc-scraper is an open-source MCP server for web search and scraping.