Skip to content
MCP server

Website Crawler — Markdown Pages, Sitemaps & RAG Provenance

Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and…

Install
Runs
Remotely
Version
1.0.4
Listing updated
4 days ago

About Website Crawler — Markdown Pages, Sitemaps & RAG Provenance

Website Crawler — Markdown Pages, Sitemaps & RAG Provenance is an MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Crawl sites to Markdown with sitemaps, globs and depth limits, obeying robots and crawl-delay

Install

Run in a terminal

claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=hermeskd/website-to-markdown'

Add to Settings → Connectors → Add custom connector

https://mcp.apify.com/?tools=hermeskd/website-to-markdown
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "website-to-markdown": {
      "url": "https://mcp.apify.com/?tools=hermeskd/website-to-markdown"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "website-to-markdown": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=hermeskd/website-to-markdown"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "website-to-markdown": {
      "serverUrl": "https://mcp.apify.com/?tools=hermeskd/website-to-markdown"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "website-to-markdown": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=hermeskd/website-to-markdown"
    }
  }
}

Website Crawler — Markdown Pages, Sitemaps & RAG Provenance FAQ

How do I add Website Crawler — Markdown Pages, Sitemaps & RAG Provenance to Claude Code?

Run this command in your terminal: claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=hermeskd/website-to-markdown'

What are the best alternatives to Website Crawler — Markdown Pages, Sitemaps & RAG Provenance?

The closest alternatives to Website Crawler — Markdown Pages, Sitemaps & RAG Provenance are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and ASTra.

Alternatives to Website Crawler — Markdown Pages, Sitemaps & RAG Provenance

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  4. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  5. ASTra

    MCP server

    Let agents read, edit, validate and undo Word .docx files with exact operations

    Runs locally
    • 5 tools
    • Freemium
    • Files and documents
  6. Search AI tools, agents and MCP servers and get install configs from inside your AI assistant.

    • 4 tools
    • Remote
    • Free
    • Developer tools