# Robots.txt & AI Crawler Checker

> Robots.txt & AI Crawler Checker is a freemium MCP server for web search and scraping. Test URLs against robots.txt, check AI crawler access and validate sitemaps. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Freemium
- **Platforms:** Web, API
- **Website:** https://robots-check.quietworklabs.workers.dev
- **MCP Registry name:** dev.workers.quietworklabs.robots-check/robots-check
- **Categories:** web search and scraping, marketing and SEO
- **Listing updated:** 2026-11-16

## About

Matching follows RFC 9309: a crawler uses the group naming its product token, otherwise the wildcard group, the longest matching path rule wins and Allow wins a tie, with * and $ as wildcards. A 4xx response for robots.txt means everything is allowed, while a 5xx or unreachable file means crawlers must assume everything is disallowed.

The checker also shows which AI crawlers a site blocks, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and reads noindex and noai tags and llms.txt before validating the XML sitemap. Crawler and agent builders can call a preflight before each fetch and receive a yes or no along with the rule that decided it.

A free web checker is available, and the API is paid per call in USDC through x402 with no key: $0.001 for the robots check and $0.002 for AI crawler and sitemap checks. The fetcher identifies itself as QuietworkRobotsCheck/1.0 and only reads robots.txt, llms.txt, the page and the sitemap.

## Features

- **RFC 9309 matching:** Group selection, longest match and wildcard rules.
- **AI crawler report:** Shows which AI bots a site blocks.
- **Preflight yes or no:** Returns the deciding rule for a URL and user agent.
- **Sitemap validation:** Checks the XML sitemap.
- **Tag and llms.txt reading:** Reads noindex, noai and llms.txt.
- **Pay per call:** $0.001 to $0.002 USDC via x402.

## Good for

- Check before fetching that your crawler is allowed
- Audit whether a site blocks GPTBot or ClaudeBot
- Debug why a page is not being crawled
- Validate a sitemap before submitting it

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://robots-check.quietworklabs.workers.dev/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "robots-check": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "robots-check": {
      "serverUrl": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "robots-check": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

## FAQ

### How are rules matched?

By RFC 9309: product token group, then wildcard group, with the longest path rule winning and Allow winning ties.

### What if robots.txt is missing?

A 4xx means everything is allowed; a 5xx or unreachable file means assume disallowed.

### What does the API cost?

$0.001 USDC for the robots check and $0.002 for crawler and sitemap checks, via x402.

### Is there a free option?

Yes, a free web checker.

### Is Robots.txt & AI Crawler Checker free?

Robots.txt & AI Crawler Checker has a free plan, with paid plans for more usage or features.

### How do I add Robots.txt & AI Crawler Checker to Claude Code?

Run this command in your terminal: claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

### What are the best alternatives to Robots.txt & AI Crawler Checker?

The closest alternatives to Robots.txt & AI Crawler Checker are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [Calven](https://ai-powered.com/mcp/calven): Calven is an MCP server for marketing and seo.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [MCP](https://ai-powered.com/mcp/mcp-mwhffe53): MCP is a freemium MCP server for marketing and seo.

---
Source: https://ai-powered.com/mcp/robots-txt-and-ai-crawler-checker
