Skip to content
MCP server

RobotsGate

Generate, validate and check robots.txt rules for AI crawlers such as GPTBot and ClaudeBot

Install
Pricing
Freemium
Platforms
API, Web
Runs
Remotely
Version
0.3.1
Listing updated
1 day ago

About RobotsGate

RobotsGate is a freemium MCP server for marketing and seo. It runs as a hosted remote server, so there is nothing to install locally.

RobotsGate keeps a registry of AI crawler user-agent tokens, each entry checked against its operator's own documentation and dated, with the list of unverified tokens visible at /api/crawlers. From that registry it can write a robots.txt, with per-group choices and per-crawler overrides where leaving a crawler out means it follows your generic rules, or validate an existing file.

Its third job is checking a live site. It fetches the site's robots.txt, limited to public http and https hosts, and shows which AI crawlers are allowed for a given path. The site's guides cover blocking GPTBot, allowing AI search while blocking AI training, and checking which bots can crawl your site.

The server has four read-only tools. The free tier allows 20 check requests and 60 generate or validate requests per minute per IP; RobotsGate Pro at $9 a month lifts that to 300 and 600 with a license key.

Install

Run in a terminal

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

Add to Settings → Connectors → Add custom connector

https://robotsgate.mike-tusa.workers.dev/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "robotsgate": {
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "robotsgate": {
      "type": "http",
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "robotsgate": {
      "serverUrl": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "robotsgate": {
      "type": "http",
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}

Tools this server provides (4)

Taken from the project documentation; the running server may differ.

list_crawlers
Lists AI crawlers in the registry.
generate_robots
Generates robots.txt rules.
validate_robots
Validates a robots.txt.
check_site
Checks which AI crawlers a live site allows.

What it does

Crawler registry
Lists AI crawler tokens with links to each operator's documentation and last-checked date.
robots.txt generator
Builds rules per crawler group with overrides.
robots.txt validator
Checks an existing file.
Live site check
Shows which AI crawlers a site's robots.txt allows for a path.
Optional Pro key
Raises rate limits for automation.
Machine-readable docs
Offers llms.txt and an OpenAPI 3.1 spec.

Good for

  • Block AI training crawlers while allowing AI search
  • Audit which AI bots can crawl a site
  • Validate a robots.txt before deploying it
  • Keep a robots.txt in step with new crawler tokens
  • Automate crawler checks across many sites

Pricing

Freemium, from $9 a month

Free tier with rate limits; Pro $9/month

Prices change. Check robotsgate.mike-tusa.workers.dev before you buy.

RobotsGate FAQ

Is it free?

Yes. The free tier has per-IP rate limits; Pro is $9 a month for higher limits.

How do I use a Pro key?

Send it as an Authorization Bearer token or an X-License-Key header; an invalid key is ignored and free limits apply.

Which hosts can it check?

Public http and https hosts only.

Where does the crawler data come from?

Each entry links to the operator's own documentation with the date it was last checked.

Is RobotsGate free?

RobotsGate has a free plan, with paid plans for more usage or features. Paid plans start at $9 a month.

How do I add RobotsGate to Claude Code?

Run this command in your terminal: claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

What are the best alternatives to RobotsGate?

The closest alternatives to RobotsGate are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

Alternatives to RobotsGate

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  4. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  5. Calven

    MCP server

    Product marketing context from Calven delivered into Claude, ChatGPT and other AI tools

    Not checked yet
    • Remote
    • Free trial
    • 3 GitHub stars
    • Marketing and SEO
  6. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping