Skip to content
MCP server

Robotstxt Ai

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and…

Runs locallyStatus and history
Install
Pricing
Open source
Runs
Locally
Version
1.0.1
Listing updated
5 days ago

About Robotstxt Ai

Robotstxt Ai is an open-source MCP server. It runs locally on your machine and is started by your AI client.

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Install

Run in a terminal

claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp

Add to claude_desktop_config.json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "robotstxt-ai": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

Using several servers? Build one config file with the MCP config generator.

Pricing

Open source

Robotstxt Ai FAQ

Is Robotstxt Ai free?

Yes. Robotstxt Ai is open source, so you can use and self-host it for free.

How do I add Robotstxt Ai to Claude Code?

Run this command in your terminal: claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp

Where this comes from

  • Description: from the official MCP Registry entry.
  • Install steps: from the official MCP Registry entry.

How AIPoweredBot checks servers