# Web Scraper & Crawler for RAG (Markdown)

> Web Scraper & Crawler for RAG (Markdown) is a paid MCP server for web search and scraping. Turn pages or whole sites into clean Markdown or JSON for LLM and RAG pipelines. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Paid
- **Platforms:** API, Web
- **Website:** https://apify.com/byteblisk/sitereader
- **Repository:** https://github.com/byteblisk/mcp-servers
- **MCP Registry name:** io.github.ByteBlisk/sitereader
- **Categories:** web search and scraping, files and documents, AI models and LLM tooling
- **Listing updated:** 2026-11-14

## About

Menus, footers and cookie banners are stripped so only the main content remains, and pages that come back empty are not charged. Modes include reading single pages, crawling a site by depth and path rules, mapping every URL of a domain, reading PDFs and watching pages for changes. A real browser is used only where a page needs JavaScript.

Ready-made feeds cover Greenhouse, Lever and Ashby job boards, Shopify stores, WordPress sites and RSS or Atom. With your own Anthropic key you can describe fields in plain words, such as every open job with title and location, and get JSON back. Pages containing text that looks like instructions to an AI are flagged with a content warning.

The actor lives on Apify and is billed pay per event at about $1.50 per 1,000 pages. In the API and MCP docs the same tool is called SiteReader.

## Features

- **Markdown extraction:** Main content as Markdown, plain text or cleaned HTML.
- **Site crawling:** Depth, path rules and page limits, plus URL discovery from sitemaps.
- **AI field extraction:** Describe fields in words; uses your own Anthropic key.
- **Ready-made feeds:** Job boards, Shopify, WordPress and RSS or Atom.
- **Prompt-injection flag:** Warns when a page contains instruction-like text.
- **Change watching:** Monitor pages for updates on a schedule.

## Good for

- Feed documentation into a vector database
- Turn a careers page into a job list
- Give an agent live web pages as clean text
- Map all URLs of a domain

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http sitereader 'https://mcp.apify.com/?tools=byteblisk/sitereader'
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://mcp.apify.com/?tools=byteblisk/sitereader
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "sitereader": {
      "url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "sitereader": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "sitereader": {
      "serverUrl": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "sitereader": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}
```

## FAQ

### How much does it cost?

From about $1.50 per 1,000 pages; empty pages are free.

### Do AI fields cost extra?

Anthropic bills the AI step to your own key; the actor charges nothing for it.

### Can it read PDFs?

Yes, PDFs are supported.

### What is SiteReader?

The name the API and MCP docs use for this same tool.

### Is Web Scraper & Crawler for RAG (Markdown) free?

No. Web Scraper & Crawler for RAG (Markdown) is a paid product.

### How do I add Web Scraper & Crawler for RAG (Markdown) to Claude Code?

Run this command in your terminal: claude mcp add --transport http sitereader 'https://mcp.apify.com/?tools=byteblisk/sitereader'

### What are the best alternatives to Web Scraper & Crawler for RAG (Markdown)?

The closest alternatives to Web Scraper & Crawler for RAG (Markdown) are Scrapling MCP Server, Firecrawl MCP Server, Claude Code Ultimate Guide, SearXNG Search and NeuralVerge.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Claude Code Ultimate Guide](https://ai-powered.com/mcp/claude-code-ultimate-guide): Claude Code Ultimate Guide is an open-source MCP server for ai models and llm tooling.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [ASTra](https://ai-powered.com/mcp/astra): ASTra is a freemium MCP server for files and documents.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [CARTESIAN ECU](https://ai-powered.com/mcp/cartesian-ecu): CARTESIAN ECU is a paid MCP server for files and documents.

---
Source: https://ai-powered.com/mcp/web-scraper-and-crawler-for-rag-markdown
