Skip to content
MCP server

doc-scraper

Crawl documentation sites into local Markdown that agents search offline

Runs locallyStatus and history
Pricing
Open source
Platforms
CLI, macOS, Windows
Runs
Locally
Version
2.27.0
Listing updated
3 days ago

About doc-scraper

doc-scraper is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.

Instead of letting an agent guess from training data or a live web search, doc-scraper stores the docs you chose on disk and serves them over MCP. Search uses BM25 full-text ranking and page reads need no network access at all.

Pointing the add command at a docs URL detects the generator behind it, such as Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs or mdBook, checks the content selector and drafts the config. Crawls respect robots.txt and sitemaps, use per-host rate limits and are resumable, and output lands as Markdown with frontmatter plus pages.jsonl, llms.txt and llms-full.txt.

It is written in Go, ships binaries for Linux, macOS and Windows, and can also be installed as an .mcpb bundle for Claude Desktop. Incremental re-crawls and watch mode keep a corpus current, with crawl history exposed through diff and freshness tools.

Install

Run in a terminal

claude mcp add doc-scraper -- npx -y https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb

Add to claude_desktop_config.json

{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "doc-scraper": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "doc-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb"
      ]
    }
  }
}

Tools this server provides (4)

Taken from the project documentation; the running server may differ.

search_docs
BM25 full-text search over the stored corpus.
read_page
Reads a stored page.
get_freshness
Reports how current a crawled site is.
diff_crawl
Compares two crawls of a site.

What it does

Offline BM25 search
search_docs and read_page answer from the stored corpus with no network.
Framework detection
Recognises 30+ docs generators and validates the content selector.
Clean Markdown output
One Markdown file per page plus pages.jsonl, llms.txt and llms-full.txt.
Polite crawling
Honours robots.txt and sitemaps with rate limits, retries and resumable runs.
Freshness tracking
Incremental re-crawls, watch mode and crawl-to-crawl diffs.
Parallel crawls
Runs many sites at once under shared limits.

Good for

  • Give a coding agent the exact docs for a library version
  • Keep an offline copy of documentation for air-gapped work
  • Detect what changed between two crawls of a docs site
  • Build an llms-full.txt file for a site

Pricing

Open source

Prices change. Check sriram-pr.github.io before you buy.

doc-scraper FAQ

Does search need an internet connection?

No. search_docs and read_page read from the stored corpus.

How do I install it?

Download a release binary, run go install, use the Docker image, or install the .mcpb bundle in Claude Desktop.

Which docs frameworks are detected?

More than 30, including Docusaurus, Starlight, Sphinx, VitePress, Material for MkDocs and mdBook.

How many MCP tools are there?

The docs reference 13 tools.

Is doc-scraper free?

Yes. doc-scraper is open source, so you can use and self-host it for free.

How do I add doc-scraper to Claude Code?

Run this command in your terminal: claude mcp add doc-scraper -- npx -y https://github.com/Sriram-PR/doc-scraper/releases/download/v2.27.0/doc-scraper.mcpb

What are the best alternatives to doc-scraper?

The closest alternatives to doc-scraper are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.

Alternatives to doc-scraper

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  3. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  4. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  5. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  6. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory