# doc-scraper

> doc-scraper is an open-source MCP server for web search and scraping. Crawl documentation sites into local Markdown that agents search offline. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, macOS, Windows, Linux, self-hosted
- **Website:** https://sriram-pr.github.io/doc-scraper/
- **Repository:** https://github.com/sriram-pr/doc-scraper
- **MCP Registry name:** io.github.Sriram-PR/doc-scraper
- **Categories:** web search and scraping, developer tools, knowledge and memory
- **Listing updated:** 2026-11-14

## About

Instead of letting an agent guess from training data or a live web search, doc-scraper stores the docs you chose on disk and serves them over MCP. Search uses BM25 full-text ranking and page reads need no network access at all.

Pointing the add command at a docs URL detects the generator behind it, such as Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs or mdBook, checks the content selector and drafts the config. Crawls respect robots.txt and sitemaps, use per-host rate limits and are resumable, and output lands as Markdown with frontmatter plus pages.jsonl, llms.txt and llms-full.txt.

It is written in Go, ships binaries for Linux, macOS and Windows, and can also be installed as an .mcpb bundle for Claude Desktop. Incremental re-crawls and watch mode keep a corpus current, with crawl history exposed through diff and freshness tools.

## Features

- **Offline BM25 search:** search_docs and read_page answer from the stored corpus with no network.
- **Framework detection:** Recognises 30+ docs generators and validates the content selector.
- **Clean Markdown output:** One Markdown file per page plus pages.jsonl, llms.txt and llms-full.txt.
- **Polite crawling:** Honours robots.txt and sitemaps with rate limits, retries and resumable runs.
- **Freshness tracking:** Incremental re-crawls, watch mode and crawl-to-crawl diffs.
- **Parallel crawls:** Runs many sites at once under shared limits.

## Good for

- Give a coding agent the exact docs for a library version
- Keep an offline copy of documentation for air-gapped work
- Detect what changed between two crawls of a docs site
- Build an llms-full.txt file for a site

## Install

### Claude Code (Terminal)

```sh
claude mcp add doc-scraper -- npx -y https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "doc-scraper": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}
```

## Tools

- `search_docs`: BM25 full-text search over the stored corpus.
- `read_page`: Reads a stored page.
- `get_freshness`: Reports how current a crawled site is.
- `diff_crawl`: Compares two crawls of a site.

## FAQ

### Does search need an internet connection?

No. search_docs and read_page read from the stored corpus.

### How do I install it?

Download a release binary, run go install, use the Docker image, or install the .mcpb bundle in Claude Desktop.

### Which docs frameworks are detected?

More than 30, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs and mdBook.

### How many MCP tools are there?

The docs reference 13 tools.

### Is doc-scraper free?

Yes. doc-scraper is open source, so you can use and self-host it for free.

### How do I add doc-scraper to Claude Code?

Run this command in your terminal: claude mcp add doc-scraper -- npx -y https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb

### What are the best alternatives to doc-scraper?

The closest alternatives to doc-scraper are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.

## Alternatives

- [REA](https://ai-powered.com/mcp/rea): REA is an open-source MCP server for developer tools.
- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Codebase Memory](https://ai-powered.com/mcp/codebase-memory): Codebase Memory is an open-source MCP server for knowledge and memory.
- [GitHub](https://ai-powered.com/mcp/github-2): GitHub is an open-source MCP server for developer tools.
- [Serena MCP: the IDE for your agent](https://ai-powered.com/mcp/serena-mcp-the-ide-for-your-agent): Serena MCP: the IDE for your agent is an open-source MCP server for developer tools.
- [XERJ](https://ai-powered.com/mcp/xerj): XERJ is an open-source MCP server for knowledge and memory.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Ouroboros](https://ai-powered.com/mcp/ouroboros): Ouroboros is an open-source MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/doc-scraper
