Skip to content
MCP server

Crawlberg

Open-source crawler that turns websites into Markdown and JSON, with a local MCP server

Runs locallyStatus and history
Pricing
Open source
Platforms
CLI, self-hosted, API
Runs
Locally
Version
1.12.3
Listing updated
2 days ago

About Crawlberg

Crawlberg is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.

Crawlberg comes from the Xberg document-intelligence project and is written around a single Rust core with 14 language bindings. Beyond fetching pages, it follows links with depth-first, breadth-first or best-first traversal, respects robots.txt and sitemaps, and throttles requests per domain.

JavaScript-heavy pages can fall back to a headless browser, and requests to loopback, private, link-local and cloud-metadata addresses are refused by default. Output includes Markdown with citations plus metadata such as JSON-LD, Open Graph, hreflang, headings and response headers. BM25 relevance scoring and URL include or exclude patterns narrow what a crawl keeps.

The registry entry ships it as a stdio container image, so an MCP client runs it locally. Managed extras such as proxy pools and scheduling sit in a separate enterprise repository rather than this one.

Install

Run in a terminal

claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.3

Add to claude_desktop_config.json

{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "crawlberg": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}

What it does

Single-page or site crawl
Scrape one URL or crawl a whole site with configurable depth, page limits and concurrency.
Markdown conversion
Pages become clean Markdown with citations and document structure.
Rich metadata
Extracts titles, links, images, JSON-LD, Open Graph, hreflang and headings.
SSRF protection
Refuses private, loopback and cloud-metadata addresses unless you opt out.
Browser fallback
Uses an optional headless browser for JavaScript-heavy single-page apps.
Many language bindings
Same core exposed to Python, Node.js, Rust, Go, Java, Ruby and others.

Good for

  • Feed a RAG pipeline with clean site content
  • Map a website's link structure and sitemap
  • Archive documentation as Markdown
  • Let an agent read pages without a hosted scraper
  • Batch-scrape hundreds of URLs concurrently

Pricing

Open source

Free to self-host; managed pipeline offered through a design partner program with pricing on application

Prices change. Check xberg.io before you buy.

Crawlberg FAQ

How does the MCP server run?

As a local stdio container image (ghcr.io/xberg-io/crawlberg); nothing is hosted for you.

What licence applies?

The Xberg site states its open-source primitives are MIT licensed and free for personal, internal and commercial use.

Does it respect robots.txt?

Yes. The README lists robots.txt compliance and sitemap discovery among its filters.

Can it crawl internal network addresses?

Not by default; private and internal addresses are refused unless you opt out through an environment variable or crawl config.

Is Crawlberg free?

Yes. Crawlberg is open source, so you can use and self-host it for free.

How do I add Crawlberg to Claude Code?

Run this command in your terminal: claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.3

What are the best alternatives to Crawlberg?

The closest alternatives to Crawlberg are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.

Alternatives to Crawlberg

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  3. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  4. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  5. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  6. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory