Web Scraper & Crawler for RAG (Markdown)
Turn pages or whole sites into clean Markdown or JSON for LLM and RAG pipelines
- Pricing
- Paid
- Platforms
- API, Web
- Runs
- Remotely
- Version
- 1.0.0
- Listing updated
- 3 days ago
About Web Scraper & Crawler for RAG (Markdown)
Web Scraper & Crawler for RAG (Markdown) is a paid MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.
Menus, footers and cookie banners are stripped so only the main content remains, and pages that come back empty are not charged. Modes include reading single pages, crawling a site by depth and path rules, mapping every URL of a domain, reading PDFs and watching pages for changes. A real browser is used only where a page needs JavaScript.
Ready-made feeds cover Greenhouse, Lever and Ashby job boards, Shopify stores, WordPress sites and RSS or Atom. With your own Anthropic key you can describe fields in plain words, such as every open job with title and location, and get JSON back. Pages containing text that looks like instructions to an AI are flagged with a content warning.
The actor lives on Apify and is billed pay per event at about $1.50 per 1,000 pages. In the API and MCP docs the same tool is called SiteReader.
Install
Run in a terminal
claude mcp add --transport http sitereader 'https://mcp.apify.com/?tools=byteblisk/sitereader'Add to Settings → Connectors → Add custom connector
https://mcp.apify.com/?tools=byteblisk/sitereaderAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"sitereader": {
"url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"sitereader": {
"type": "http",
"url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"sitereader": {
"serverUrl": "https://mcp.apify.com/?tools=byteblisk/sitereader"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"sitereader": {
"type": "http",
"url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
}
}
}What it does
- Markdown extraction
- Main content as Markdown, plain text or cleaned HTML.
- Site crawling
- Depth, path rules and page limits, plus URL discovery from sitemaps.
- AI field extraction
- Describe fields in words; uses your own Anthropic key.
- Ready-made feeds
- Job boards, Shopify, WordPress and RSS or Atom.
- Prompt-injection flag
- Warns when a page contains instruction-like text.
- Change watching
- Monitor pages for updates on a schedule.
Good for
- Feed documentation into a vector database
- Turn a careers page into a job list
- Give an agent live web pages as clean text
- Map all URLs of a domain
Pricing
Paid
Pay per event on Apify, from about $1.50 per 1,000 pages; Apify free plan includes monthly credit.
Prices change. Check apify.com before you buy.
Web Scraper & Crawler for RAG (Markdown) FAQ
How much does it cost?
From about $1.50 per 1,000 pages; empty pages are free.
Do AI fields cost extra?
Anthropic bills the AI step to your own key; the actor charges nothing for it.
Can it read PDFs?
Yes, PDFs are supported.
What is SiteReader?
The name the API and MCP docs use for this same tool.
Is Web Scraper & Crawler for RAG (Markdown) free?
No. Web Scraper & Crawler for RAG (Markdown) is a paid product.
How do I add Web Scraper & Crawler for RAG (Markdown) to Claude Code?
Run this command in your terminal: claude mcp add --transport http sitereader 'https://mcp.apify.com/?tools=byteblisk/sitereader'
What are the best alternatives to Web Scraper & Crawler for RAG (Markdown)?
The closest alternatives to Web Scraper & Crawler for RAG (Markdown) are Scrapling MCP Server, Firecrawl MCP Server, Claude Code Ultimate Guide, SearXNG Search and NeuralVerge.
Alternatives to Web Scraper & Crawler for RAG (Markdown)

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Claude Code Ultimate Guide
MCP serverMCP server for searching the Claude Code Ultimate Guide from any MCP client
Runs locally- Open source
- 6.5K GitHub stars
- AI models and LLM tooling

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping

ASTra
MCP serverLet agents read, edit, validate and undo Word .docx files with exact operations
Runs locally- 5 tools
- Freemium
- Files and documents