# Website to Markdown Crawler

> Website to Markdown Crawler is a paid MCP server for web search and scraping. Crawl a site into clean, main-content Markdown for RAG, with linked documents converted. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Paid
- **Platforms:** API
- **Website:** https://apify.com/prometheus-maker/website-to-markdown-crawler
- **Repository:** https://github.com/prometheus-maker/apify-tools-examples
- **MCP Registry name:** io.github.prometheus-maker/website-to-markdown
- **Categories:** web search and scraping, files and documents, AI models and LLM tooling
- **Listing updated:** 2026-11-14

## About

Start from one URL and the crawler follows the site's links, keeping only each page's main content as Markdown. Menus, cookie banners, sidebars and footers are dropped, while headings, lists, tables, code blocks and links stay. PDF, Word, PowerPoint and Excel files linked from the pages are converted too, so a help centre comes back with its manuals.

The author reports a test on 45 pages from 8 sites including MDN, GOV.UK and react.dev: about 3% of the kept text repeated from page to page, against 29% repeated boilerplate when whole pages are converted. Output is one item per page and one per converted file, each with title, language, word count and depth. Setting chunkSize also splits items at paragraph boundaries, with the headings each chunk sits under.

It uses no browser, which keeps it fast, so script-rendered pages are a likely weak spot. It runs on Apify with an Apify-hosted MCP endpoint.

## Features

- **Main content only:** Strips navigation, banners and footers so less repeated text reaches your model.
- **Linked files converted:** Turns linked PDF, Word, PowerPoint and Excel files into Markdown.
- **Optional chunking:** Splits output into pieces with the headings each piece belongs to.
- **Structure preserved:** Keeps headings, lists, tables, code blocks and links.
- **No browser:** Fetches pages over plain HTTP, so crawls are quick.
- **Page and document items:** Returns separate item types for pages and converted files, with source URLs.

## Good for

- Feed a documentation site into a vector database
- Let an agent read a whole docs section in one call
- Export a site's text for migration or translation
- Collect public reports with their linked PDFs as text

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler'
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "website-to-markdown": {
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "website-to-markdown": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "website-to-markdown": {
      "serverUrl": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "website-to-markdown": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler"
    }
  }
}
```

## FAQ

### Does it handle PDFs linked from pages?

Yes. Linked PDF, Word, PowerPoint and Excel files are converted to Markdown as separate document items.

### Does it use a browser?

No, which is why it is fast, according to the author.

### How is it priced?

Pay per page on Apify. The store lists from $1.00 per 1,000 pages.

### Can output be chunked for RAG?

Yes. Set chunkSize and each item also gets chunks split at paragraph boundaries.

### Is Website to Markdown Crawler free?

No. Website to Markdown Crawler is a paid product.

### How do I add Website to Markdown Crawler to Claude Code?

Run this command in your terminal: claude mcp add --transport http website-to-markdown 'https://mcp.apify.com/?tools=prometheus-maker/website-to-markdown-crawler'

### What are the best alternatives to Website to Markdown Crawler?

The closest alternatives to Website to Markdown Crawler are Scrapling MCP Server, Firecrawl MCP Server, Claude Code Ultimate Guide, SearXNG Search and NeuralVerge.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Claude Code Ultimate Guide](https://ai-powered.com/mcp/claude-code-ultimate-guide): Claude Code Ultimate Guide is an open-source MCP server for ai models and llm tooling.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [ASTra](https://ai-powered.com/mcp/astra): ASTra is a freemium MCP server for files and documents.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [CARTESIAN ECU](https://ai-powered.com/mcp/cartesian-ecu): CARTESIAN ECU is a paid MCP server for files and documents.

---
Source: https://ai-powered.com/mcp/website-to-markdown-crawler
