Crawl Readiness
Check which AI crawlers can read a site and generate llms.txt, robots.txt and schema
- Pricing
- Freemium
- Platforms
- API, CLI, macOS
- Runs
- Locally
- Version
- 0.2.9
- Listing updated
- 2 days ago
About Crawl Readiness
Crawl Readiness is a freemium MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.
Where most robots.txt checkers stop at syntax, this server scores a site from 0 to 100 against 60 AI crawlers, including ChatGPT, Claude, Perplexity and Google AI, and returns per-crawler status with a prioritised fix list. A separate permission check asks whether a page may be used for search, answers, training or agent action. It reads robots.txt, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and page directives. Any reservation wins, and an unreadable source gives unknown rather than allowed. The result is a dated receipt signed with Ed25519.
There are nine tools. Four audits and the permission check work anonymously (25 permission calls a day per network without a key). Three generators, for llms.txt, robots.txt and JSON-LD, need a free API key, as does reading Answer Analysis trends. The robots.txt generator offers four presets, from allow-all to block-all. The content parity check fetches a page as Chrome, GPTBot, ClaudeBot and PerplexityBot to spot JS-only shells and cloaking.
The free key gives 50 checks a month. Paid plans add monitoring, bulk checks and white-label reports.
Install
Run in a terminal
claude mcp add crawl-readiness --env CRAWL_READINESS_API_KEY='<CRAWL_READINESS_API_KEY>' -- npx -y crawl-readiness-mcpAdd to claude_desktop_config.json
{
"mcpServers": {
"crawl-readiness": {
"command": "npx",
"args": [
"-y",
"crawl-readiness-mcp"
],
"env": {
"CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
}
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"crawl-readiness": {
"command": "npx",
"args": [
"-y",
"crawl-readiness-mcp"
],
"env": {
"CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
}
}
}
}Add to .vscode/mcp.json
{
"servers": {
"crawl-readiness": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"crawl-readiness-mcp"
],
"env": {
"CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
}
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"crawl-readiness": {
"command": "npx",
"args": [
"-y",
"crawl-readiness-mcp"
],
"env": {
"CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
}
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"crawl-readiness": {
"command": "npx",
"args": [
"-y",
"crawl-readiness-mcp"
],
"env": {
"CRAWL_READINESS_API_KEY": "<CRAWL_READINESS_API_KEY>"
}
}
}
}Configuration
CRAWL_READINESS_API_KEYFree API key from https://www.crawlreadiness.com/dashboard. The four audit tools and check_permission work without it; the three generators need it, and get_monitor_trend needs one on a plan with Answer Analyses.CRAWL_READINESS_API_BASEOverride the API base URL. Only useful for testing against a preview deployment; defaults to https://www.crawlreadiness.com.
Tools this server provides (9)
Taken from the project documentation; the running server may differ.
- check_ai_readiness
- Scores a site 0-100 against AI crawlers with a fix list.
- validate_schema
- Audits JSON-LD structured data on a URL.
- validate_robots
- Line-by-line robots.txt audit.
- check_content_parity
- Compares what humans and AI crawlers see.
- check_permission
- Checks whether a page may be used for search, answers, training or agent action.
- generate_llms_txt
- Crawls a site and produces llms.txt.
- generate_robots_txt
- Produces an AI-crawler-aware robots.txt from presets.
- generate_schema
- Produces Organization, WebSite or Article JSON-LD.
- get_monitor_trend
- Reads Answer Analysis results over time.
What it does
- Crawler readiness score
- Scores a site 0-100 against 60 AI crawlers with a prioritised fix list.
- Signed permission receipts
- Answers whether a use is allowed and returns an Ed25519-signed receipt.
- Content parity check
- Compares what Chrome, GPTBot, ClaudeBot and PerplexityBot each receive.
- JSON-LD validation
- Audits structured data against 20+ common types.
- File generators
- Writes llms.txt, robots.txt presets and Organization or Article JSON-LD.
- Answer Analysis trend
- Reads whether AI search named your brand or competitors over time.
Good for
- Find out whether GPTBot or ClaudeBot is blocked on a site
- Generate llms.txt from a sitemap
- Audit a robots.txt for orphan rules and wildcard traps
- Validate JSON-LD before publishing
- Check whether a page permits training use
Pricing
Freemium, from $39 a month
Free key: 50 checks a month; Monitor $39/mo, Studio $79/mo, Agency $199/mo, audit $99 once
Prices change. Check crawlreadiness.com before you buy.
Crawl Readiness FAQ
Which tools need an API key?
The four audit tools and check_permission work anonymously. The three generators and get_monitor_trend need a key.
Is the API key free?
A free key from the dashboard gives 50 checks a month with no credit card.
What are the paid options?
The pricing page lists Monitor at $39 a month, Studio at $79, Agency at $199 and a one-time $99 audit.
Which clients does it work with?
Claude Desktop, Cursor, Cline, Windsurf and Continue, and it ships as an npm package and an .mcpb bundle.
Is Crawl Readiness free?
Crawl Readiness has a free plan, with paid plans for more usage or features. Paid plans start at $39 a month.
How do I add Crawl Readiness to Claude Code?
Run this command in your terminal: claude mcp add crawl-readiness --env CRAWL_READINESS_API_KEY='<CRAWL_READINESS_API_KEY>' -- npx -y crawl-readiness-mcp
What are the best alternatives to Crawl Readiness?
The closest alternatives to Crawl Readiness are REA, Scrapling MCP Server, Codebase Memory, GitHub and Serena MCP: the IDE for your agent.
Alternatives to Crawl Readiness

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Codebase Memory
MCP serverLocal MCP server that builds a code knowledge graph for AI coding agents
Runs locally- Open source
- 48.7K GitHub stars
- Knowledge and memory

GitHub
MCP serverOfficial MCP server connecting AI assistants to GitHub repos, issues and workflows
Not checked yet- Remote
- Open source
- 34.5K GitHub stars
- Developer tools

Serena MCP: the IDE for your agent
MCP serverMCP toolkit giving coding agents symbol-level code retrieval and editing
Runs locally- Open source
- 31.5K GitHub stars
- Developer tools

XERJ
MCP serverLocal search engine for AI agents with cited answers over your code and docs
Runs locally- Open source
- 19.2K GitHub stars
- Knowledge and memory