Crawlberg
Open-source crawler that turns websites into Markdown and JSON, with a local MCP server
- Pricing
- Open source
- Platforms
- CLI, self-hosted, API
- Runs
- Locally
- Version
- 1.12.3
- Listing updated
- 2 days ago
About Crawlberg
Crawlberg is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.
Crawlberg comes from the Xberg document-intelligence project and is written around a single Rust core with 14 language bindings. Beyond fetching pages, it follows links with depth-first, breadth-first or best-first traversal, respects robots.txt and sitemaps, and throttles requests per domain.
JavaScript-heavy pages can fall back to a headless browser, and requests to loopback, private, link-local and cloud-metadata addresses are refused by default. Output includes Markdown with citations plus metadata such as JSON-LD, Open Graph, hreflang, headings and response headers. BM25 relevance scoring and URL include or exclude patterns narrow what a crawl keeps.
The registry entry ships it as a stdio container image, so an MCP client runs it locally. Managed extras such as proxy pools and scheduling sit in a separate enterprise repository rather than this one.
Install
Run in a terminal
claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.3Add to claude_desktop_config.json
{
"mcpServers": {
"crawlberg": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/xberg-io/crawlberg:1.12.3"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"crawlberg": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/xberg-io/crawlberg:1.12.3"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"crawlberg": {
"type": "stdio",
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/xberg-io/crawlberg:1.12.3"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"crawlberg": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/xberg-io/crawlberg:1.12.3"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"crawlberg": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/xberg-io/crawlberg:1.12.3"
]
}
}
}What it does
- Single-page or site crawl
- Scrape one URL or crawl a whole site with configurable depth, page limits and concurrency.
- Markdown conversion
- Pages become clean Markdown with citations and document structure.
- Rich metadata
- Extracts titles, links, images, JSON-LD, Open Graph, hreflang and headings.
- SSRF protection
- Refuses private, loopback and cloud-metadata addresses unless you opt out.
- Browser fallback
- Uses an optional headless browser for JavaScript-heavy single-page apps.
- Many language bindings
- Same core exposed to Python, Node.js, Rust, Go, Java, Ruby and others.
Good for
- Feed a RAG pipeline with clean site content
- Map a website's link structure and sitemap
- Archive documentation as Markdown
- Let an agent read pages without a hosted scraper
- Batch-scrape hundreds of URLs concurrently
Pricing
Open source
Free to self-host; managed pipeline offered through a design partner program with pricing on application
Prices change. Check xberg.io before you buy.
Crawlberg FAQ
How does the MCP server run?
As a local stdio container image (ghcr.io/xberg-io/crawlberg); nothing is hosted for you.
What licence applies?
The Xberg site states its open-source primitives are MIT licensed and free for personal, internal and commercial use.
Does it respect robots.txt?
Yes. The README lists robots.txt compliance and sitemap discovery among its filters.
Can it crawl internal network addresses?
Not by default; private and internal addresses are refused unless you opt out through an environment variable or crawl config.
Is Crawlberg free?
Yes. Crawlberg is open source, so you can use and self-host it for free.
How do I add Crawlberg to Claude Code?
Run this command in your terminal: claude mcp add crawlberg -- docker run -i --rm ghcr.io/xberg-io/crawlberg:1.12.3
What are the best alternatives to Crawlberg?
The closest alternatives to Crawlberg are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.
Alternatives to Crawlberg

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Codebase Memory
MCP serverLocal MCP server that builds a code knowledge graph for AI coding agents
Runs locally- Open source
- 48.7K GitHub stars
- Knowledge and memory

GitHub
MCP serverOfficial MCP server connecting AI assistants to GitHub repos, issues and workflows
Not checked yet- Remote
- Open source
- 34.5K GitHub stars
- Developer tools

Serena MCP: the IDE for your agent
MCP serverMCP toolkit giving coding agents symbol-level code retrieval and editing
Runs locally- Open source
- 31.5K GitHub stars
- Developer tools

XERJ
MCP serverLocal search engine for AI agents with cited answers over your code and docs
Runs locally- Open source
- 19.2K GitHub stars
- Knowledge and memory