RobotsGate
Generate, validate and check robots.txt rules for AI crawlers such as GPTBot and ClaudeBot
- Pricing
- Freemium
- Platforms
- API, Web
- Runs
- Remotely
- Version
- 0.3.1
- Listing updated
- 1 day ago
About RobotsGate
RobotsGate is a freemium MCP server for marketing and seo. It runs as a hosted remote server, so there is nothing to install locally.
RobotsGate keeps a registry of AI crawler user-agent tokens, each entry checked against its operator's own documentation and dated, with the list of unverified tokens visible at /api/crawlers. From that registry it can write a robots.txt, with per-group choices and per-crawler overrides where leaving a crawler out means it follows your generic rules, or validate an existing file.
Its third job is checking a live site. It fetches the site's robots.txt, limited to public http and https hosts, and shows which AI crawlers are allowed for a given path. The site's guides cover blocking GPTBot, allowing AI search while blocking AI training, and checking which bots can crawl your site.
The server has four read-only tools. The free tier allows 20 check requests and 60 generate or validate requests per minute per IP; RobotsGate Pro at $9 a month lifts that to 300 and 600 with a license key.
Install
Run in a terminal
claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcpAdd to Settings → Connectors → Add custom connector
https://robotsgate.mike-tusa.workers.dev/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"robotsgate": {
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"robotsgate": {
"type": "http",
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"robotsgate": {
"serverUrl": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"robotsgate": {
"type": "http",
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Tools this server provides (4)
Taken from the project documentation; the running server may differ.
- list_crawlers
- Lists AI crawlers in the registry.
- generate_robots
- Generates robots.txt rules.
- validate_robots
- Validates a robots.txt.
- check_site
- Checks which AI crawlers a live site allows.
What it does
- Crawler registry
- Lists AI crawler tokens with links to each operator's documentation and last-checked date.
- robots.txt generator
- Builds rules per crawler group with overrides.
- robots.txt validator
- Checks an existing file.
- Live site check
- Shows which AI crawlers a site's robots.txt allows for a path.
- Optional Pro key
- Raises rate limits for automation.
- Machine-readable docs
- Offers llms.txt and an OpenAPI 3.1 spec.
Good for
- Block AI training crawlers while allowing AI search
- Audit which AI bots can crawl a site
- Validate a robots.txt before deploying it
- Keep a robots.txt in step with new crawler tokens
- Automate crawler checks across many sites
Pricing
Freemium, from $9 a month
Free tier with rate limits; Pro $9/month
Prices change. Check robotsgate.mike-tusa.workers.dev before you buy.
RobotsGate FAQ
Is it free?
Yes. The free tier has per-IP rate limits; Pro is $9 a month for higher limits.
How do I use a Pro key?
Send it as an Authorization Bearer token or an X-License-Key header; an invalid key is ignored and free limits apply.
Which hosts can it check?
Public http and https hosts only.
Where does the crawler data come from?
Each entry links to the operator's own documentation with the date it was last checked.
Is RobotsGate free?
RobotsGate has a free plan, with paid plans for more usage or features. Paid plans start at $9 a month.
How do I add RobotsGate to Claude Code?
Run this command in your terminal: claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp
What are the best alternatives to RobotsGate?
The closest alternatives to RobotsGate are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.
Alternatives to RobotsGate

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Google Ads + Meta Ads + SEO/GEO MCP
MCP serverHosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat
- 8 tools
- Remote
- Freemium
- 8.1K GitHub stars
- Marketing and SEO

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

Calven
MCP serverProduct marketing context from Calven delivered into Claude, ChatGPT and other AI tools
Not checked yet- Remote
- Free trial
- 3 GitHub stars
- Marketing and SEO

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping