# Crawlberg

> Crawlberg is an open-source MCP server for web search and scraping. Open-source crawler that turns websites into Markdown and JSON, with a local MCP server. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, self-hosted, API, Linux, macOS, Windows
- **Website:** https://xberg.io
- **Repository:** https://github.com/xberg-io/crawlberg
- **MCP Registry name:** io.github.xberg-io/crawlberg
- **Categories:** web search and scraping, developer tools
- **Listing updated:** 2026-11-15

## About

Crawlberg comes from the Xberg document-intelligence project and is written around a single Rust core with 14 language bindings. Beyond fetching pages, it follows links with depth-first, breadth-first or best-first traversal, respects robots.txt and sitemaps, and throttles requests per domain.

JavaScript-heavy pages can fall back to a headless browser, and requests to loopback, private, link-local and cloud-metadata addresses are refused by default. Output includes Markdown with citations plus metadata such as JSON-LD, Open Graph, hreflang, headings and response headers. BM25 relevance scoring and URL include or exclude patterns narrow what a crawl keeps.

The registry entry ships it as a stdio container image, so an MCP client runs it locally. Managed extras such as proxy pools and scheduling sit in a separate enterprise repository rather than this one.

## Features

- **Single-page or site crawl:** Scrape one URL or crawl a whole site with configurable depth, page limits and concurrency.
- **Markdown conversion:** Pages become clean Markdown with citations and document structure.
- **Rich metadata:** Extracts titles, links, images, JSON-LD, Open Graph, hreflang and headings.
- **SSRF protection:** Refuses private, loopback and cloud-metadata addresses unless you opt out.
- **Browser fallback:** Uses an optional headless browser for JavaScript-heavy single-page apps.
- **Many language bindings:** Same core exposed to Python, Node.js, Rust, Go, Java, Ruby and others.

## Good for

- Feed a RAG pipeline with clean site content
- Map a website's link structure and sitemap
- Archive documentation as Markdown
- Let an agent read pages without a hosted scraper
- Batch-scrape hundreds of URLs concurrently

## Install

### Claude Code (Terminal)

```sh
claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.3
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "crawlberg": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.12.3"
      ]
    }
  }
}
```

## FAQ

### How does the MCP server run?

As a local stdio container image (ghcr.io/xberg-io/crawlberg); nothing is hosted for you.

### What licence applies?

The Xberg site states its open-source primitives are MIT licensed and free for personal, internal and commercial use.

### Does it respect robots.txt?

Yes. The README lists robots.txt compliance and sitemap discovery among its filters.

### Can it crawl internal network addresses?

Not by default; private and internal addresses are refused unless you opt out through an environment variable or crawl config.

### Is Crawlberg free?

Yes. Crawlberg is open source, so you can use and self-host it for free.

### How do I add Crawlberg to Claude Code?

Run this command in your terminal: claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.3

### What are the best alternatives to Crawlberg?

The closest alternatives to Crawlberg are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.

## Alternatives

- [REA](https://ai-powered.com/mcp/rea): REA is an open-source MCP server for developer tools.
- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Codebase Memory](https://ai-powered.com/mcp/codebase-memory): Codebase Memory is an open-source MCP server for knowledge and memory.
- [GitHub](https://ai-powered.com/mcp/github-2): GitHub is an open-source MCP server for developer tools.
- [Serena MCP: the IDE for your agent](https://ai-powered.com/mcp/serena-mcp-the-ide-for-your-agent): Serena MCP: the IDE for your agent is an open-source MCP server for developer tools.
- [XERJ](https://ai-powered.com/mcp/xerj): XERJ is an open-source MCP server for knowledge and memory.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Ouroboros](https://ai-powered.com/mcp/ouroboros): Ouroboros is an open-source MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/crawlberg
