Robots.txt & AI Crawler Checker
Test URLs against robots.txt, check AI crawler access and validate sitemaps
- Pricing
- Freemium
- Platforms
- Web, API
- Runs
- Remotely
- Version
- 1.0.0
- Listing updated
- 1 day ago
About Robots.txt & AI Crawler Checker
Robots.txt & AI Crawler Checker is a freemium MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.
Matching follows RFC 9309: a crawler uses the group naming its product token, otherwise the wildcard group, the longest matching path rule wins and Allow wins a tie, with * and $ as wildcards. A 4xx response for robots.txt means everything is allowed, while a 5xx or unreachable file means crawlers must assume everything is disallowed.
The checker also shows which AI crawlers a site blocks, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and reads noindex and noai tags and llms.txt before validating the XML sitemap. Crawler and agent builders can call a preflight before each fetch and receive a yes or no along with the rule that decided it.
A free web checker is available, and the API is paid per call in USDC through x402 with no key: $0.001 for the robots check and $0.002 for AI crawler and sitemap checks. The fetcher identifies itself as QuietworkRobotsCheck/1.0 and only reads robots.txt, llms.txt, the page and the sitemap.
Install
Run in a terminal
claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcpAdd to Settings → Connectors → Add custom connector
https://robots-check.quietworklabs.workers.dev/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"robots-check": {
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"robots-check": {
"type": "http",
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"robots-check": {
"serverUrl": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"robots-check": {
"type": "http",
"url": "https://robots-check.quietworklabs.workers.dev/mcp"
}
}
}What it does
- RFC 9309 matching
- Group selection, longest match and wildcard rules.
- AI crawler report
- Shows which AI bots a site blocks.
- Preflight yes or no
- Returns the deciding rule for a URL and user agent.
- Sitemap validation
- Checks the XML sitemap.
- Tag and llms.txt reading
- Reads noindex, noai and llms.txt.
- Pay per call
- $0.001 to $0.002 USDC via x402.
Good for
- Check before fetching that your crawler is allowed
- Audit whether a site blocks GPTBot or ClaudeBot
- Debug why a page is not being crawled
- Validate a sitemap before submitting it
Pricing
Freemium
Free web checker; API $0.001-$0.002 USDC per call via x402
Prices change. Check robots-check.quietworklabs.workers.dev before you buy.
Robots.txt & AI Crawler Checker FAQ
How are rules matched?
By RFC 9309: product token group, then wildcard group, with the longest path rule winning and Allow winning ties.
What if robots.txt is missing?
A 4xx means everything is allowed; a 5xx or unreachable file means assume disallowed.
What does the API cost?
$0.001 USDC for the robots check and $0.002 for crawler and sitemap checks, via x402.
Is there a free option?
Yes, a free web checker.
Is Robots.txt & AI Crawler Checker free?
Robots.txt & AI Crawler Checker has a free plan, with paid plans for more usage or features.
How do I add Robots.txt & AI Crawler Checker to Claude Code?
Run this command in your terminal: claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp
What are the best alternatives to Robots.txt & AI Crawler Checker?
The closest alternatives to Robots.txt & AI Crawler Checker are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.
Alternatives to Robots.txt & AI Crawler Checker

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Google Ads + Meta Ads + SEO/GEO MCP
MCP serverHosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat
- 8 tools
- Remote
- Freemium
- 8.1K GitHub stars
- Marketing and SEO

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

Calven
MCP serverProduct marketing context from Calven delivered into Claude, ChatGPT and other AI tools
Not checked yet- Remote
- Free trial
- 3 GitHub stars
- Marketing and SEO

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping