Website to Markdown Crawler
Crawl a site into clean, main-content Markdown for RAG, with linked documents converted
- Pricing
- Paid
- Platforms
- API
- Runs
- Remotely
- Version
- 1.0.0
- Listing updated
- 3 days ago
About Website to Markdown Crawler
Website to Markdown Crawler is a paid MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.
Start from one URL and the crawler follows the site's links, keeping only each page's main content as Markdown. Menus, cookie banners, sidebars and footers are dropped, while headings, lists, tables, code blocks and links stay. PDF, Word, PowerPoint and Excel files linked from the pages are converted too, so a help centre comes back with its manuals.
The author reports a test on 45 pages from 8 sites including MDN, GOV.UK and react.dev: about 3% of the kept text repeated from page to page, against 29% repeated boilerplate when whole pages are converted. Output is one item per page and one per converted file, each with title, language, word count and depth. Setting chunkSize also splits items at paragraph boundaries, with the headings each chunk sits under.
It uses no browser, which keeps it fast, so script-rendered pages are a likely weak spot. It runs on Apify with an Apify-hosted MCP endpoint.
Install
Run in a terminal
claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler'Add to Settings → Connectors → Add custom connector
https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawlerAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"website-to-markdown": {
"url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"website-to-markdown": {
"type": "http",
"url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"website-to-markdown": {
"serverUrl": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"website-to-markdown": {
"type": "http",
"url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
}
}
}What it does
- Main content only
- Strips navigation, banners and footers so less repeated text reaches your model.
- Linked files converted
- Turns linked PDF, Word, PowerPoint and Excel files into Markdown.
- Optional chunking
- Splits output into pieces with the headings each piece belongs to.
- Structure preserved
- Keeps headings, lists, tables, code blocks and links.
- No browser
- Fetches pages over plain HTTP, so crawls are quick.
- Page and document items
- Returns separate item types for pages and converted files, with source URLs.
Good for
- Feed a documentation site into a vector database
- Let an agent read a whole docs section in one call
- Export a site's text for migration or translation
- Collect public reports with their linked PDFs as text
Pricing
Paid
Pay per use on Apify; from $1.00 per 1,000 pages saved, no subscription
Prices change. Check apify.com before you buy.
Website to Markdown Crawler FAQ
Does it handle PDFs linked from pages?
Yes. Linked PDF, Word, PowerPoint and Excel files are converted to Markdown as separate document items.
Does it use a browser?
No, which is why it is fast, according to the author.
How is it priced?
Pay per page on Apify. The store lists from $1.00 per 1,000 pages.
Can output be chunked for RAG?
Yes. Set chunkSize and each item also gets chunks split at paragraph boundaries.
Is Website to Markdown Crawler free?
No. Website to Markdown Crawler is a paid product.
How do I add Website to Markdown Crawler to Claude Code?
Run this command in your terminal: claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler'
What are the best alternatives to Website to Markdown Crawler?
The closest alternatives to Website to Markdown Crawler are Scrapling MCP Server, Firecrawl MCP Server, Claude Code Ultimate Guide, SearXNG Search and NeuralVerge.
Alternatives to Website to Markdown Crawler

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Claude Code Ultimate Guide
MCP serverMCP server for searching the Claude Code Ultimate Guide from any MCP client
Runs locally- Open source
- 6.5K GitHub stars
- AI models and LLM tooling

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping

ASTra
MCP serverLet agents read, edit, validate and undo Word .docx files with exact operations
Runs locally- 5 tools
- Freemium
- Files and documents