MCP server
Robotstxt Ai
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and…
Runs locallyStatus and history
- Pricing
- Open source
- Runs
- Locally
- Version
- 1.0.1
- Listing updated
- 5 days ago
About Robotstxt Ai
Robotstxt Ai is an open-source MCP server. It runs locally on your machine and is started by your AI client.
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
Install
Run in a terminal
claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcpAdd to claude_desktop_config.json
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:
Add to ~/.cursor/mcp.json
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:
Add to .vscode/mcp.json
{
"servers": {
"robotstxt-ai": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}Using several servers? Build one config file with the MCP config generator.
Pricing
Open source
Robotstxt Ai FAQ
Is Robotstxt Ai free?
Yes. Robotstxt Ai is open source, so you can use and self-host it for free.
How do I add Robotstxt Ai to Claude Code?
Run this command in your terminal: claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp
Where this comes from
- Description: from the official MCP Registry entry.
- Install steps: from the official MCP Registry entry.