Skip to content
MCP server

Unrendered: AI Crawler Checker

Shows which page text AI crawlers miss because it only appears after JavaScript runs

Runs locallyStatus and history
Install
Pricing
Open source
Platforms
CLI, Chrome extension
Runs
Locally
Version
0.1.4
Listing updated
2 days ago

About Unrendered: AI Crawler Checker

Unrendered: AI Crawler Checker is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.

AI crawlers such as GPTBot and ClaudeBot generally fetch raw HTML without running JavaScript, so text added by scripts is invisible to them. This server compares a page's raw HTML with the version Chrome renders and reports the gap. It is the same analysis as the Unrendered Chrome extension, bundled and run inside your own installed Chrome.

analyze_page returns the share of main content present in the raw HTML, the biggest text blocks that exist only after JavaScript, and differences in title, meta description, canonical, hreflang, robots and JSON-LD. It says why no figure is given when the raw request hits a challenge page or error. check_ai_bots reports, for each bot, what robots.txt allows and how the site answers that bot's user-agent.

It runs locally over stdio and needs Node 20 or later and Google Chrome. Private and loopback addresses are refused by default because URLs come from a model. The project is AGPL-3.0 and sends nothing anywhere except to the site being checked.

Install

Run in a terminal

claude mcp add unrendered-ai-crawler-checker -- npx -y unrendered-mcp

Add to claude_desktop_config.json

{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "unrendered-ai-crawler-checker": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "unrendered-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "unrendered-mcp"
      ]
    }
  }
}

Tools this server provides (2)

Taken from the project documentation; the running server may differ.

analyze_page
Compares raw and rendered HTML for a URL and returns coverage, missing blocks and field differences.
check_ai_bots
Reports robots.txt verdicts and how the site answers each AI bot's user-agent.

What it does

Raw versus rendered coverage
Measures how much main content is in the raw HTML versus added by JavaScript.
Missing text blocks
Lists the largest blocks that appear only after scripts run.
Head field differences
Compares title, canonical, hreflang, robots meta and JSON-LD in both versions.
Per-bot checks
Shows robots.txt verdicts and how the site answers each AI bot's user-agent.
Private-host guard
Refuses loopback and private addresses unless an environment variable allows them.

Good for

  • Audit whether a JavaScript-heavy site is readable by AI crawlers
  • Check that a noindex tag is not removed by scripts
  • Find out whether robots.txt blocks GPTBot or ClaudeBot
  • Review a list of URLs with an assistant before a launch

Pricing

Open source

Free, AGPL-3.0

Prices change. Check missiongrowth.io before you buy.

Unrendered: AI Crawler Checker FAQ

What does it need?

Node 20 or later and Google Chrome; no browser is downloaded.

Does it send data anywhere?

No. The README says nothing is sent except requests to the site you analyse, and nothing is stored.

Can it test a local dev server?

Yes, after setting UNRENDERED_ALLOW_PRIVATE_HOSTS=1.

What licence is it under?

AGPL-3.0.

Is Unrendered: AI Crawler Checker free?

Yes. Unrendered: AI Crawler Checker is open source, so you can use and self-host it for free.

How do I add Unrendered: AI Crawler Checker to Claude Code?

Run this command in your terminal: claude mcp add unrendered-ai-crawler-checker -- npx -y unrendered-mcp

What are the best alternatives to Unrendered: AI Crawler Checker?

The closest alternatives to Unrendered: AI Crawler Checker are Browser Use, Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP and SearXNG Search.

Alternatives to Unrendered: AI Crawler Checker

  1. Browser Use

    MCP server

    Browser agents and cloud browsers for AI, with an MCP server for web tasks

    • 7 tools
    • Remote
    • Freemium
    • 121K GitHub stars
    • Browser automation
  2. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  3. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  4. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  5. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  6. Oya Browser

    MCP server

    HIPAA-oriented agentic browser with replayable playbooks, reachable over MCP

    • 32 tools
    • Remote
    • Freemium
    • 349 GitHub stars
    • Browser automation