Skip to content
MCP server

Web Scraper & Crawler for RAG (Markdown)

Turn pages or whole sites into clean Markdown or JSON for LLM and RAG pipelines

Install
Pricing
Paid
Platforms
API, Web
Runs
Remotely
Version
1.0.0
Listing updated
3 days ago

About Web Scraper & Crawler for RAG (Markdown)

Web Scraper & Crawler for RAG (Markdown) is a paid MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Menus, footers and cookie banners are stripped so only the main content remains, and pages that come back empty are not charged. Modes include reading single pages, crawling a site by depth and path rules, mapping every URL of a domain, reading PDFs and watching pages for changes. A real browser is used only where a page needs JavaScript.

Ready-made feeds cover Greenhouse, Lever and Ashby job boards, Shopify stores, WordPress sites and RSS or Atom. With your own Anthropic key you can describe fields in plain words, such as every open job with title and location, and get JSON back. Pages containing text that looks like instructions to an AI are flagged with a content warning.

The actor lives on Apify and is billed pay per event at about $1.50 per 1,000 pages. In the API and MCP docs the same tool is called SiteReader.

Install

Run in a terminal

claude mcp add --transport http sitereader 'https://mcp.apify.com/?tools=byteblisk/sitereader'

Add to Settings → Connectors → Add custom connector

https://mcp.apify.com/?tools=byteblisk/sitereader
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "sitereader": {
      "url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "sitereader": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "sitereader": {
      "serverUrl": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "sitereader": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=byteblisk/sitereader"
    }
  }
}

What it does

Markdown extraction
Main content as Markdown, plain text or cleaned HTML.
Site crawling
Depth, path rules and page limits, plus URL discovery from sitemaps.
AI field extraction
Describe fields in words; uses your own Anthropic key.
Ready-made feeds
Job boards, Shopify, WordPress and RSS or Atom.
Prompt-injection flag
Warns when a page contains instruction-like text.
Change watching
Monitor pages for updates on a schedule.

Good for

  • Feed documentation into a vector database
  • Turn a careers page into a job list
  • Give an agent live web pages as clean text
  • Map all URLs of a domain

Pricing

Paid

Pay per event on Apify, from about $1.50 per 1,000 pages; Apify free plan includes monthly credit.

Prices change. Check apify.com before you buy.

Web Scraper & Crawler for RAG (Markdown) FAQ

How much does it cost?

From about $1.50 per 1,000 pages; empty pages are free.

Do AI fields cost extra?

Anthropic bills the AI step to your own key; the actor charges nothing for it.

Can it read PDFs?

Yes, PDFs are supported.

What is SiteReader?

The name the API and MCP docs use for this same tool.

Is Web Scraper & Crawler for RAG (Markdown) free?

No. Web Scraper & Crawler for RAG (Markdown) is a paid product.

How do I add Web Scraper & Crawler for RAG (Markdown) to Claude Code?

Run this command in your terminal: claude mcp add --transport http sitereader 'https://mcp.apify.com/?tools=byteblisk/sitereader'

What are the best alternatives to Web Scraper & Crawler for RAG (Markdown)?

The closest alternatives to Web Scraper & Crawler for RAG (Markdown) are Scrapling MCP Server, Firecrawl MCP Server, Claude Code Ultimate Guide, SearXNG Search and NeuralVerge.

Alternatives to Web Scraper & Crawler for RAG (Markdown)

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. MCP server for searching the Claude Code Ultimate Guide from any MCP client

    Runs locally
    • Open source
    • 6.5K GitHub stars
    • AI models and LLM tooling
  4. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  5. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  6. ASTra

    MCP server

    Let agents read, edit, validate and undo Word .docx files with exact operations

    Runs locally
    • 5 tools
    • Freemium
    • Files and documents