# RobotsGate

> RobotsGate is a freemium MCP server for marketing and seo. Generate, validate and check robots.txt rules for AI crawlers such as GPTBot and ClaudeBot. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Freemium
- **Starting price:** $9 a month
- **Platforms:** API, Web
- **Website:** https://robotsgate.mike-tusa.workers.dev
- **MCP Registry name:** dev.workers.mike-tusa.robotsgate/robotsgate
- **Categories:** marketing and SEO, web search and scraping
- **Listing updated:** 2026-11-16

## About

RobotsGate keeps a registry of AI crawler user-agent tokens, each entry checked against its operator's own documentation and dated, with the list of unverified tokens visible at /api/crawlers. From that registry it can write a robots.txt, with per-group choices and per-crawler overrides where leaving a crawler out means it follows your generic rules, or validate an existing file.

Its third job is checking a live site. It fetches the site's robots.txt, limited to public http and https hosts, and shows which AI crawlers are allowed for a given path. The site's guides cover blocking GPTBot, allowing AI search while blocking AI training, and checking which bots can crawl your site.

The server has four read-only tools. The free tier allows 20 check requests and 60 generate or validate requests per minute per IP; RobotsGate Pro at $9 a month lifts that to 300 and 600 with a license key.

## Features

- **Crawler registry:** Lists AI crawler tokens with links to each operator's documentation and last-checked date.
- **robots.txt generator:** Builds rules per crawler group with overrides.
- **robots.txt validator:** Checks an existing file.
- **Live site check:** Shows which AI crawlers a site's robots.txt allows for a path.
- **Optional Pro key:** Raises rate limits for automation.
- **Machine-readable docs:** Offers llms.txt and an OpenAPI 3.1 spec.

## Good for

- Block AI training crawlers while allowing AI search
- Audit which AI bots can crawl a site
- Validate a robots.txt before deploying it
- Keep a robots.txt in step with new crawler tokens
- Automate crawler checks across many sites

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://robotsgate.mike-tusa.workers.dev/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "robotsgate": {
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "robotsgate": {
      "type": "http",
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "robotsgate": {
      "serverUrl": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "robotsgate": {
      "type": "http",
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}
```

## Tools

- `list_crawlers`: Lists AI crawlers in the registry.
- `generate_robots`: Generates robots.txt rules.
- `validate_robots`: Validates a robots.txt.
- `check_site`: Checks which AI crawlers a live site allows.

## FAQ

### Is it free?

Yes. The free tier has per-IP rate limits; Pro is $9 a month for higher limits.

### How do I use a Pro key?

Send it as an Authorization Bearer token or an X-License-Key header; an invalid key is ignored and free limits apply.

### Which hosts can it check?

Public http and https hosts only.

### Where does the crawler data come from?

Each entry links to the operator's own documentation with the date it was last checked.

### Is RobotsGate free?

RobotsGate has a free plan, with paid plans for more usage or features. Paid plans start at $9 a month.

### How do I add RobotsGate to Claude Code?

Run this command in your terminal: claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

### What are the best alternatives to RobotsGate?

The closest alternatives to RobotsGate are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [Calven](https://ai-powered.com/mcp/calven): Calven is an MCP server for marketing and seo.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [MCP](https://ai-powered.com/mcp/mcp-mwhffe53): MCP is a freemium MCP server for marketing and seo.

---
Source: https://ai-powered.com/mcp/robotsgate
