doc-scraper
Crawl documentation sites into local Markdown that agents search offline
- Pricing
- Open source
- Platforms
- CLI, macOS, Windows
- Runs
- Locally
- Version
- 2.27.0
- Listing updated
- 3 days ago
About doc-scraper
doc-scraper is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.
Instead of letting an agent guess from training data or a live web search, doc-scraper stores the docs you chose on disk and serves them over MCP. Search uses BM25 full-text ranking and page reads need no network access at all.
Pointing the add command at a docs URL detects the generator behind it, such as Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs or mdBook, checks the content selector and drafts the config. Crawls respect robots.txt and sitemaps, use per-host rate limits and are resumable, and output lands as Markdown with frontmatter plus pages.jsonl, llms.txt and llms-full.txt.
It is written in Go, ships binaries for Linux, macOS and Windows, and can also be installed as an .mcpb bundle for Claude Desktop. Incremental re-crawls and watch mode keep a corpus current, with crawl history exposed through diff and freshness tools.
Install
Run in a terminal
claude mcp add doc-scraper -- npx -y https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpbAdd to claude_desktop_config.json
{
"mcpServers": {
"doc-scraper": {
"command": "npx",
"args": [
"-y",
"https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"doc-scraper": {
"command": "npx",
"args": [
"-y",
"https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"doc-scraper": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"doc-scraper": {
"command": "npx",
"args": [
"-y",
"https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"doc-scraper": {
"command": "npx",
"args": [
"-y",
"https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
]
}
}
}Tools this server provides (4)
Taken from the project documentation; the running server may differ.
- search_docs
- BM25 full-text search over the stored corpus.
- read_page
- Reads a stored page.
- get_freshness
- Reports how current a crawled site is.
- diff_crawl
- Compares two crawls of a site.
What it does
- Offline BM25 search
- search_docs and read_page answer from the stored corpus with no network.
- Framework detection
- Recognises 30+ docs generators and validates the content selector.
- Clean Markdown output
- One Markdown file per page plus pages.jsonl, llms.txt and llms-full.txt.
- Polite crawling
- Honours robots.txt and sitemaps with rate limits, retries and resumable runs.
- Freshness tracking
- Incremental re-crawls, watch mode and crawl-to-crawl diffs.
- Parallel crawls
- Runs many sites at once under shared limits.
Good for
- Give a coding agent the exact docs for a library version
- Keep an offline copy of documentation for air-gapped work
- Detect what changed between two crawls of a docs site
- Build an llms-full.txt file for a site
Pricing
Open source
Prices change. Check sriram-pr.github.io before you buy.
doc-scraper FAQ
Does search need an internet connection?
No. search_docs and read_page read from the stored corpus.
How do I install it?
Download a release binary, run go install, use the Docker image, or install the .mcpb bundle in Claude Desktop.
Which docs frameworks are detected?
More than 30, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs and mdBook.
How many MCP tools are there?
The docs reference 13 tools.
Is doc-scraper free?
Yes. doc-scraper is open source, so you can use and self-host it for free.
How do I add doc-scraper to Claude Code?
Run this command in your terminal: claude mcp add doc-scraper -- npx -y https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb
What are the best alternatives to doc-scraper?
The closest alternatives to doc-scraper are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.
Alternatives to doc-scraper

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Codebase Memory
MCP serverLocal MCP server that builds a code knowledge graph for AI coding agents
Runs locally- Open source
- 48.7K GitHub stars
- Knowledge and memory

GitHub
MCP serverOfficial MCP server connecting AI assistants to GitHub repos, issues and workflows
Not checked yet- Remote
- Open source
- 34.5K GitHub stars
- Developer tools

Serena MCP: the IDE for your agent
MCP serverMCP toolkit giving coding agents symbol-level code retrieval and editing
Runs locally- Open source
- 31.5K GitHub stars
- Developer tools

XERJ
MCP serverLocal search engine for AI agents with cited answers over your code and docs
Runs locally- Open source
- 19.2K GitHub stars
- Knowledge and memory