Skip to content

Crawlberg vs doc-scraper

Crawlberg is an open-source MCP server for web search and scraping. doc-scraper is an open-source MCP server for web search and scraping.

  • Crawlberg is best for feed a rag pipeline with clean site content.
  • doc-scraper is best for give a coding agent the exact docs for a library version.
What it is
Crawlberg is an open-source MCP server for web search and scraping.
doc-scraper is an open-source MCP server for web search and scraping.
Best for
Feed a RAG pipeline with clean site content
Give a coding agent the exact docs for a library version
Kind
MCP server
MCP server
Pricing
Open source
Open source
Platforms
CLI, self-hosted, API, Linux, macOS, Windows
CLI, macOS, Windows, Linux, self-hosted
Runs
Locally
Locally
Open source
Good for
Feed a RAG pipeline with clean site content; Map a website's link structure and sitemap; Archive documentation as Markdown
Give a coding agent the exact docs for a library version; Keep an offline copy of documentation for air-gapped work; Detect what changed between two crawls of a docs site

What Crawlberg does

  • Single-page or site crawl

    Scrape one URL or crawl a whole site with configurable depth, page limits and concurrency.

  • Markdown conversion

    Pages become clean Markdown with citations and document structure.

  • Rich metadata

    Extracts titles, links, images, JSON-LD, Open Graph, hreflang and headings.

  • SSRF protection

    Refuses private, loopback and cloud-metadata addresses unless you opt out.

  • Browser fallback

    Uses an optional headless browser for JavaScript-heavy single-page apps.

  • Many language bindings

    Same core exposed to Python, Node.js, Rust, Go, Java, Ruby and others.

What doc-scraper does

  • Offline BM25 search

    search_docs and read_page answer from the stored corpus with no network.

  • Framework detection

    Recognises 30+ docs generators and validates the content selector.

  • Clean Markdown output

    One Markdown file per page plus pages.jsonl, llms.txt and llms-full.txt.

  • Polite crawling

    Honours robots.txt and sitemaps with rate limits, retries and resumable runs.

  • Freshness tracking

    Incremental re-crawls, watch mode and crawl-to-crawl diffs.

  • Parallel crawls

    Runs many sites at once under shared limits.

Crawlberg vs doc-scraper: questions

Is Crawlberg or doc-scraper cheaper?

Both are open source. Compare current plans on each official site.

What is the difference between Crawlberg and doc-scraper?

Crawlberg is an open-source MCP server for web search and scraping. doc-scraper is an open-source MCP server for web search and scraping.

Keep comparing