# Unrendered: AI Crawler Checker

> Unrendered: AI Crawler Checker is an open-source MCP server for web search and scraping. Shows which page text AI crawlers miss because it only appears after JavaScript runs. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, Chrome extension
- **Website:** https://missiongrowth.io/tools/unrendered
- **Repository:** https://github.com/missiongrowth/unrendered-ai-crawler-checker-mcp/tree/HEAD/mcp
- **MCP Registry name:** io.github.missiongrowth/unrendered-ai-crawler-checker
- **Categories:** web search and scraping, marketing and SEO, browser automation
- **Listing updated:** 2026-11-15

## About

AI crawlers such as GPTBot and ClaudeBot generally fetch raw HTML without running JavaScript, so text added by scripts is invisible to them. This server compares a page's raw HTML with the version Chrome renders and reports the gap. It is the same analysis as the Unrendered Chrome extension, bundled and run inside your own installed Chrome.

analyze_page returns the share of main content present in the raw HTML, the biggest text blocks that exist only after JavaScript, and differences in title, meta description, canonical, hreflang, robots and JSON-LD. It says why no figure is given when the raw request hits a challenge page or error. check_ai_bots reports, for each bot, what robots.txt allows and how the site answers that bot's user-agent.

It runs locally over stdio and needs Node 20 or later and Google Chrome. Private and loopback addresses are refused by default because URLs come from a model. The project is AGPL-3.0 and sends nothing anywhere except to the site being checked.

## Features

- **Raw versus rendered coverage:** Measures how much main content is in the raw HTML versus added by JavaScript.
- **Missing text blocks:** Lists the largest blocks that appear only after scripts run.
- **Head field differences:** Compares title, canonical, hreflang, robots meta and JSON-LD in both versions.
- **Per-bot checks:** Shows robots.txt verdicts and how the site answers each AI bot's user-agent.
- **Private-host guard:** Refuses loopback and private addresses unless an environment variable allows them.

## Good for

- Audit whether a JavaScript-heavy site is readable by AI crawlers
- Check that a noindex tag is not removed by scripts
- Find out whether robots.txt blocks GPTBot or ClaudeBot
- Review a list of URLs with an assistant before a launch

## Install

### Claude Code (Terminal)

```sh
claude mcp add unrendered-ai-crawler-checker -- npx -y unrendered-mcp
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "unrendered-ai-crawler-checker": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}
```

## Tools

- `analyze_page`: Compares raw and rendered HTML for a URL and returns coverage, missing blocks and field differences.
- `check_ai_bots`: Reports robots.txt verdicts and how the site answers each AI bot's user-agent.

## FAQ

### What does it need?

Node 20 or later and Google Chrome; no browser is downloaded.

### Does it send data anywhere?

No. The README says nothing is sent except requests to the site you analyse, and nothing is stored.

### Can it test a local dev server?

Yes, after setting UNRENDERED_ALLOW_PRIVATE_HOSTS=1.

### What licence is it under?

AGPL-3.0.

### Is Unrendered: AI Crawler Checker free?

Yes. Unrendered: AI Crawler Checker is open source, so you can use and self-host it for free.

### How do I add Unrendered: AI Crawler Checker to Claude Code?

Run this command in your terminal: claude mcp add unrendered-ai-crawler-checker -- npx -y unrendered-mcp

### What are the best alternatives to Unrendered: AI Crawler Checker?

The closest alternatives to Unrendered: AI Crawler Checker are Browser Use, Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP and SearXNG Search.

## Alternatives

- [Browser Use](https://ai-powered.com/mcp/browser-use): Browser Use is a freemium MCP server for browser automation.
- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [Oya Browser](https://ai-powered.com/mcp/oya-browser): Oya Browser is a freemium MCP server for browser automation.
- [Mantis](https://ai-powered.com/mcp/mantis-2): Mantis is an open-source MCP server for browser automation.
- [Calven](https://ai-powered.com/mcp/calven): Calven is an MCP server for marketing and seo.

---
Source: https://ai-powered.com/mcp/unrendered-ai-crawler-checker
