Skip to content
MCP server

Crawl Readiness

Check which AI crawlers can read a site and generate llms.txt, robots.txt and schema

Runs locallyStatus and history
Pricing
Freemium
Platforms
API, CLI, macOS
Runs
Locally
Version
0.2.9
Listing updated
2 days ago

About Crawl Readiness

Crawl Readiness is a freemium MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.

Where most robots.txt checkers stop at syntax, this server scores a site from 0 to 100 against 60 AI crawlers, including ChatGPT, Claude, Perplexity and Google AI, and returns per-crawler status with a prioritised fix list. A separate permission check asks whether a page may be used for search, answers, training or agent action. It reads robots.txt, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and page directives. Any reservation wins, and an unreadable source gives unknown rather than allowed. The result is a dated receipt signed with Ed25519.

There are nine tools. Four audits and the permission check work anonymously (25 permission calls a day per network without a key). Three generators, for llms.txt, robots.txt and JSON-LD, need a free API key, as does reading Answer Analysis trends. The robots.txt generator offers four presets, from allow-all to block-all. The content parity check fetches a page as Chrome, GPTBot, ClaudeBot and PerplexityBot to spot JS-only shells and cloaking.

The free key gives 50 checks a month. Paid plans add monitoring, bulk checks and white-label reports.

Install

Run in a terminal

claude mcp add crawl-readiness --env CRAWL_READINESS_API_KEY='<CRAWL_READINESS_API_KEY>' -- npx -y crawl-readiness-mcp

Add to claude_desktop_config.json

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}

Add to .vscode/mcp.json

{
  "servers": {
    "crawl-readiness": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ],
      "env": {
        "CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
      }
    }
  }
}

Configuration

  • CRAWL_READINESS_API_KEYFree API key from https://www.crawlreadiness.com/dashboard. The four audit tools and check_permission work without it; the three generators need it, and get_monitor_trend needs one on a plan with Answer Analyses.
  • CRAWL_READINESS_API_BASEOverride the API base URL. Only useful for testing against a preview deployment; defaults to https://www.crawlreadiness.com.

Tools this server provides (9)

Taken from the project documentation; the running server may differ.

check_ai_readiness
Scores a site 0-100 against AI crawlers with a fix list.
validate_schema
Audits JSON-LD structured data on a URL.
validate_robots
Line-by-line robots.txt audit.
check_content_parity
Compares what humans and AI crawlers see.
check_permission
Checks whether a page may be used for search, answers, training or agent action.
generate_llms_txt
Crawls a site and produces llms.txt.
generate_robots_txt
Produces an AI-crawler-aware robots.txt from presets.
generate_schema
Produces Organization, WebSite or Article JSON-LD.
get_monitor_trend
Reads Answer Analysis results over time.

What it does

Crawler readiness score
Scores a site 0-100 against 60 AI crawlers with a prioritised fix list.
Signed permission receipts
Answers whether a use is allowed and returns an Ed25519-signed receipt.
Content parity check
Compares what Chrome, GPTBot, ClaudeBot and PerplexityBot each receive.
JSON-LD validation
Audits structured data against 20+ common types.
File generators
Writes llms.txt, robots.txt presets and Organization or Article JSON-LD.
Answer Analysis trend
Reads whether AI search named your brand or competitors over time.

Good for

  • Find out whether GPTBot or ClaudeBot is blocked on a site
  • Generate llms.txt from a sitemap
  • Audit a robots.txt for orphan rules and wildcard traps
  • Validate JSON-LD before publishing
  • Check whether a page permits training use

Pricing

Freemium, from $39 a month

Free key: 50 checks a month; Monitor $39/mo, Studio $79/mo, Agency $199/mo, audit $99 once

Prices change. Check crawlreadiness.com before you buy.

Crawl Readiness FAQ

Which tools need an API key?

The four audit tools and check_permission work anonymously. The three generators and get_monitor_trend need a key.

Is the API key free?

A free key from the dashboard gives 50 checks a month with no credit card.

What are the paid options?

The pricing page lists Monitor at $39 a month, Studio at $79, Agency at $199 and a one-time $99 audit.

Which clients does it work with?

Claude Desktop, Cursor, Cline, Windsurf and Continue, and it ships as an npm package and an .mcpb bundle.

Is Crawl Readiness free?

Crawl Readiness has a free plan, with paid plans for more usage or features. Paid plans start at $39 a month.

How do I add Crawl Readiness to Claude Code?

Run this command in your terminal: claude mcp add crawl-readiness --env CRAWL_READINESS_API_KEY='<CRAWL_READINESS_API_KEY>' -- npx -y crawl-readiness-mcp

What are the best alternatives to Crawl Readiness?

The closest alternatives to Crawl Readiness are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.

Alternatives to Crawl Readiness

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  3. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  4. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  5. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  6. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory