Skip to content
MCP server

Robots.txt & AI Crawler Checker

Test URLs against robots.txt, check AI crawler access and validate sitemaps

Install
Pricing
Freemium
Platforms
Web, API
Runs
Remotely
Version
1.0.0
Listing updated
1 day ago

About Robots.txt & AI Crawler Checker

Robots.txt & AI Crawler Checker is a freemium MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Matching follows RFC 9309: a crawler uses the group naming its product token, otherwise the wildcard group, the longest matching path rule wins and Allow wins a tie, with * and $ as wildcards. A 4xx response for robots.txt means everything is allowed, while a 5xx or unreachable file means crawlers must assume everything is disallowed.

The checker also shows which AI crawlers a site blocks, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and reads noindex and noai tags and llms.txt before validating the XML sitemap. Crawler and agent builders can call a preflight before each fetch and receive a yes or no along with the rule that decided it.

A free web checker is available, and the API is paid per call in USDC through x402 with no key: $0.001 for the robots check and $0.002 for AI crawler and sitemap checks. The fetcher identifies itself as QuietworkRobotsCheck/1.0 and only reads robots.txt, llms.txt, the page and the sitemap.

Install

Run in a terminal

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

Add to Settings → Connectors → Add custom connector

https://robots-check.quietworklabs.workers.dev/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "robots-check": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "robots-check": {
      "serverUrl": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "robots-check": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

What it does

RFC 9309 matching
Group selection, longest match and wildcard rules.
AI crawler report
Shows which AI bots a site blocks.
Preflight yes or no
Returns the deciding rule for a URL and user agent.
Sitemap validation
Checks the XML sitemap.
Tag and llms.txt reading
Reads noindex, noai and llms.txt.
Pay per call
$0.001 to $0.002 USDC via x402.

Good for

  • Check before fetching that your crawler is allowed
  • Audit whether a site blocks GPTBot or ClaudeBot
  • Debug why a page is not being crawled
  • Validate a sitemap before submitting it

Pricing

Freemium

Free web checker; API $0.001-$0.002 USDC per call via x402

Prices change. Check robots-check.quietworklabs.workers.dev before you buy.

Robots.txt & AI Crawler Checker FAQ

How are rules matched?

By RFC 9309: product token group, then wildcard group, with the longest path rule winning and Allow winning ties.

What if robots.txt is missing?

A 4xx means everything is allowed; a 5xx or unreachable file means assume disallowed.

What does the API cost?

$0.001 USDC for the robots check and $0.002 for crawler and sitemap checks, via x402.

Is there a free option?

Yes, a free web checker.

Is Robots.txt & AI Crawler Checker free?

Robots.txt & AI Crawler Checker has a free plan, with paid plans for more usage or features.

How do I add Robots.txt & AI Crawler Checker to Claude Code?

Run this command in your terminal: claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

What are the best alternatives to Robots.txt & AI Crawler Checker?

The closest alternatives to Robots.txt & AI Crawler Checker are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

Alternatives to Robots.txt & AI Crawler Checker

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  4. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  5. Calven

    MCP server

    Product marketing context from Calven delivered into Claude, ChatGPT and other AI tools

    Not checked yet
    • Remote
    • Free trial
    • 3 GitHub stars
    • Marketing and SEO
  6. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping