Skip to content
MCP server

Website to Markdown Crawler

Crawl a site into clean, main-content Markdown for RAG, with linked documents converted

Install
Pricing
Paid
Platforms
API
Runs
Remotely
Version
1.0.0
Listing updated
3 days ago

About Website to Markdown Crawler

Website to Markdown Crawler is a paid MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Start from one URL and the crawler follows the site's links, keeping only each page's main content as Markdown. Menus, cookie banners, sidebars and footers are dropped, while headings, lists, tables, code blocks and links stay. PDF, Word, PowerPoint and Excel files linked from the pages are converted too, so a help centre comes back with its manuals.

The author reports a test on 45 pages from 8 sites including MDN, GOV.UK and react.dev: about 3% of the kept text repeated from page to page, against 29% repeated boilerplate when whole pages are converted. Output is one item per page and one per converted file, each with title, language, word count and depth. Setting chunkSize also splits items at paragraph boundaries, with the headings each chunk sits under.

It uses no browser, which keeps it fast, so script-rendered pages are a likely weak spot. It runs on Apify with an Apify-hosted MCP endpoint.

Install

Run in a terminal

claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler'

Add to Settings → Connectors → Add custom connector

https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "website-to-markdown": {
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "website-to-markdown": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "website-to-markdown": {
      "serverUrl": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "website-to-markdown": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}

What it does

Main content only
Strips navigation, banners and footers so less repeated text reaches your model.
Linked files converted
Turns linked PDF, Word, PowerPoint and Excel files into Markdown.
Optional chunking
Splits output into pieces with the headings each piece belongs to.
Structure preserved
Keeps headings, lists, tables, code blocks and links.
No browser
Fetches pages over plain HTTP, so crawls are quick.
Page and document items
Returns separate item types for pages and converted files, with source URLs.

Good for

  • Feed a documentation site into a vector database
  • Let an agent read a whole docs section in one call
  • Export a site's text for migration or translation
  • Collect public reports with their linked PDFs as text

Pricing

Paid

Pay per use on Apify; from $1.00 per 1,000 pages saved, no subscription

Prices change. Check apify.com before you buy.

Website to Markdown Crawler FAQ

Does it handle PDFs linked from pages?

Yes. Linked PDF, Word, PowerPoint and Excel files are converted to Markdown as separate document items.

Does it use a browser?

No, which is why it is fast, according to the author.

How is it priced?

Pay per page on Apify. The store lists from $1.00 per 1,000 pages.

Can output be chunked for RAG?

Yes. Set chunkSize and each item also gets chunks split at paragraph boundaries.

Is Website to Markdown Crawler free?

No. Website to Markdown Crawler is a paid product.

How do I add Website to Markdown Crawler to Claude Code?

Run this command in your terminal: claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler'

What are the best alternatives to Website to Markdown Crawler?

The closest alternatives to Website to Markdown Crawler are Scrapling MCP Server, Firecrawl MCP Server, Claude Code Ultimate Guide, SearXNG Search and NeuralVerge.

Alternatives to Website to Markdown Crawler

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. MCP server for searching the Claude Code Ultimate Guide from any MCP client

    Runs locally
    • Open source
    • 6.5K GitHub stars
    • AI models and LLM tooling
  4. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  5. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  6. ASTra

    MCP server

    Let agents read, edit, validate and undo Word .docx files with exact operations

    Runs locally
    • 5 tools
    • Freemium
    • Files and documents