Unrendered: AI Crawler Checker
Shows which page text AI crawlers miss because it only appears after JavaScript runs
- Pricing
- Open source
- Platforms
- CLI, Chrome extension
- Runs
- Locally
- Version
- 0.1.4
- Listing updated
- 2 days ago
About Unrendered: AI Crawler Checker
Unrendered: AI Crawler Checker is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.
AI crawlers such as GPTBot and ClaudeBot generally fetch raw HTML without running JavaScript, so text added by scripts is invisible to them. This server compares a page's raw HTML with the version Chrome renders and reports the gap. It is the same analysis as the Unrendered Chrome extension, bundled and run inside your own installed Chrome.
analyze_page returns the share of main content present in the raw HTML, the biggest text blocks that exist only after JavaScript, and differences in title, meta description, canonical, hreflang, robots and JSON-LD. It says why no figure is given when the raw request hits a challenge page or error. check_ai_bots reports, for each bot, what robots.txt allows and how the site answers that bot's user-agent.
It runs locally over stdio and needs Node 20 or later and Google Chrome. Private and loopback addresses are refused by default because URLs come from a model. The project is AGPL-3.0 and sends nothing anywhere except to the site being checked.
Install
Run in a terminal
claude mcp add unrendered-ai-crawler-checker -- npx -y unrendered-mcpAdd to claude_desktop_config.json
{
"mcpServers": {
"unrendered-ai-crawler-checker": {
"command": "npx",
"args": [
"-y",
"unrendered-mcp"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"unrendered-ai-crawler-checker": {
"command": "npx",
"args": [
"-y",
"unrendered-mcp"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"unrendered-ai-crawler-checker": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"unrendered-mcp"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"unrendered-ai-crawler-checker": {
"command": "npx",
"args": [
"-y",
"unrendered-mcp"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"unrendered-ai-crawler-checker": {
"command": "npx",
"args": [
"-y",
"unrendered-mcp"
]
}
}
}Tools this server provides (2)
Taken from the project documentation; the running server may differ.
- analyze_page
- Compares raw and rendered HTML for a URL and returns coverage, missing blocks and field differences.
- check_ai_bots
- Reports robots.txt verdicts and how the site answers each AI bot's user-agent.
What it does
- Raw versus rendered coverage
- Measures how much main content is in the raw HTML versus added by JavaScript.
- Missing text blocks
- Lists the largest blocks that appear only after scripts run.
- Head field differences
- Compares title, canonical, hreflang, robots meta and JSON-LD in both versions.
- Per-bot checks
- Shows robots.txt verdicts and how the site answers each AI bot's user-agent.
- Private-host guard
- Refuses loopback and private addresses unless an environment variable allows them.
Good for
- Audit whether a JavaScript-heavy site is readable by AI crawlers
- Check that a noindex tag is not removed by scripts
- Find out whether robots.txt blocks GPTBot or ClaudeBot
- Review a list of URLs with an assistant before a launch
Pricing
Open source
Free, AGPL-3.0
Prices change. Check missiongrowth.io before you buy.
Unrendered: AI Crawler Checker FAQ
What does it need?
Node 20 or later and Google Chrome; no browser is downloaded.
Does it send data anywhere?
No. The README says nothing is sent except requests to the site you analyse, and nothing is stored.
Can it test a local dev server?
Yes, after setting UNRENDERED_ALLOW_PRIVATE_HOSTS=1.
What licence is it under?
AGPL-3.0.
Is Unrendered: AI Crawler Checker free?
Yes. Unrendered: AI Crawler Checker is open source, so you can use and self-host it for free.
How do I add Unrendered: AI Crawler Checker to Claude Code?
Run this command in your terminal: claude mcp add unrendered-ai-crawler-checker -- npx -y unrendered-mcp
What are the best alternatives to Unrendered: AI Crawler Checker?
The closest alternatives to Unrendered: AI Crawler Checker are Browser Use, Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP and SearXNG Search.
Alternatives to Unrendered: AI Crawler Checker

Browser Use
MCP serverBrowser agents and cloud browsers for AI, with an MCP server for web tasks
- 7 tools
- Remote
- Freemium
- 121K GitHub stars
- Browser automation

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Google Ads + Meta Ads + SEO/GEO MCP
MCP serverHosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat
- 8 tools
- Remote
- Freemium
- 8.1K GitHub stars
- Marketing and SEO

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

Oya Browser
MCP serverHIPAA-oriented agentic browser with replayable playbooks, reachable over MCP
- 32 tools
- Remote
- Freemium
- 349 GitHub stars
- Browser automation