# Crawl Readiness

> Crawl Readiness is a freemium MCP server for web search and scraping. Check which AI crawlers can read a site and generate llms.txt, robots.txt and schema. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Freemium
- **Starting price:** $39 a month
- **Platforms:** API, CLI, macOS, Windows, Linux
- **Website:** https://www.crawlreadiness.com/mcp
- **Repository:** https://github.com/grubbomatic/crawl-readiness-mcp
- **MCP Registry name:** io.github.Grubbomatic/crawl-readiness
- **Categories:** web search and scraping, marketing and SEO, developer tools
- **Listing updated:** 2026-11-14

## About

Where most robots.txt checkers stop at syntax, this server scores a site from 0 to 100 against 60 AI crawlers, including ChatGPT, Claude, Perplexity and Google AI, and returns per-crawler status with a prioritised fix list. A separate permission check asks whether a page may be used for search, answers, training or agent action. It reads robots.txt, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and page directives. Any reservation wins, and an unreadable source gives unknown rather than allowed. The result is a dated receipt signed with Ed25519.

There are nine tools. Four audits and the permission check work anonymously (25 permission calls a day per network without a key). Three generators, for llms.txt, robots.txt and JSON-LD, need a free API key, as does reading Answer Analysis trends. The robots.txt generator offers four presets, from allow-all to block-all. The content parity check fetches a page as Chrome, GPTBot, ClaudeBot and PerplexityBot to spot JS-only shells and cloaking.

The free key gives 50 checks a month. Paid plans add monitoring, bulk checks and white-label reports.

## Features

- **Crawler readiness score:** Scores a site 0-100 against 60 AI crawlers with a prioritised fix list.
- **Signed permission receipts:** Answers whether a use is allowed and returns an Ed25519-signed receipt.
- **Content parity check:** Compares what Chrome, GPTBot, ClaudeBot and PerplexityBot each receive.
- **JSON-LD validation:** Audits structured data against 20+ common types.
- **File generators:** Writes llms.txt, robots.txt presets and Organization or Article JSON-LD.
- **Answer Analysis trend:** Reads whether AI search named your brand or competitors over time.

## Good for

- Find out whether GPTBot or ClaudeBot is blocked on a site
- Generate llms.txt from a sitemap
- Audit a robots.txt for orphan rules and wildcard traps
- Validate JSON-LD before publishing
- Check whether a page permits training use

## Install

### Claude Code (Terminal)

```sh
claude mcp add crawl-readiness --env CRAWL_READINESS_API_KEY='<CRAWL_READINESS_API_KEY>' -- npx -y crawl-readiness-mcp
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "crawl-readiness": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}
```

## Tools

- `check_ai_readiness`: Scores a site 0-100 against AI crawlers with a fix list.
- `validate_schema`: Audits JSON-LD structured data on a URL.
- `validate_robots`: Line-by-line robots.txt audit.
- `check_content_parity`: Compares what humans and AI crawlers see.
- `check_permission`: Checks whether a page may be used for search, answers, training or agent action.
- `generate_llms_txt`: Crawls a site and produces llms.txt.
- `generate_robots_txt`: Produces an AI-crawler-aware robots.txt from presets.
- `generate_schema`: Produces Organization, WebSite or Article JSON-LD.
- `get_monitor_trend`: Reads Answer Analysis results over time.

## FAQ

### Which tools need an API key?

The four audit tools and check_permission work anonymously. The three generators and get_monitor_trend need a key.

### Is the API key free?

A free key from the dashboard gives 50 checks a month with no credit card.

### What are the paid options?

The pricing page lists Monitor at $39 a month, Studio at $79, Agency at $199 and a one-time $99 audit.

### Which clients does it work with?

Claude Desktop, Cursor, Cline, Windsurf and Continue, and it ships as an npm package and an .mcpb bundle.

### Is Crawl Readiness free?

Crawl Readiness has a free plan, with paid plans for more usage or features. Paid plans start at $39 a month.

### How do I add Crawl Readiness to Claude Code?

Run this command in your terminal: claude mcp add crawl-readiness --env CRAWL_READINESS_API_KEY='<CRAWL_READINESS_API_KEY>' -- npx -y crawl-readiness-mcp

### What are the best alternatives to Crawl Readiness?

The closest alternatives to Crawl Readiness are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.

## Alternatives

- [REA](https://ai-powered.com/mcp/rea): REA is an open-source MCP server for developer tools.
- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Codebase Memory](https://ai-powered.com/mcp/codebase-memory): Codebase Memory is an open-source MCP server for knowledge and memory.
- [GitHub](https://ai-powered.com/mcp/github-2): GitHub is an open-source MCP server for developer tools.
- [Serena MCP: the IDE for your agent](https://ai-powered.com/mcp/serena-mcp-the-ide-for-your-agent): Serena MCP: the IDE for your agent is an open-source MCP server for developer tools.
- [XERJ](https://ai-powered.com/mcp/xerj): XERJ is an open-source MCP server for knowledge and memory.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Ouroboros](https://ai-powered.com/mcp/ouroboros): Ouroboros is an open-source MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/crawl-readiness
