Robots Txt Ai Crawler Audit
Checks robots.txt line by line and says what each AI crawler token actually controls…
- Pricing
- Open source
- Runs
- Locally
- Version
- 0.1.3
- Listing updated
- 5 days ago
Similar MCP servers
About Robots Txt Ai Crawler Audit
Robots Txt Ai Crawler Audit is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
Install
Run in a terminal
claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-auditAdd to claude_desktop_config.json
{
"mcpServers": {
"robots-txt-ai-crawler-audit": {
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"robots-txt-ai-crawler-audit": {
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"robots-txt-ai-crawler-audit": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"robots-txt-ai-crawler-audit": {
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"robots-txt-ai-crawler-audit": {
"command": "npx",
"args": [
"-y",
"@readystack/robots-txt-ai-crawler-audit"
]
}
}
}Using several servers? Build one config file with the MCP config generator.
Pricing
Open source
Robots Txt Ai Crawler Audit FAQ
Is Robots Txt Ai Crawler Audit free?
Yes. Robots Txt Ai Crawler Audit is open source, so you can use and self-host it for free.
How do I add Robots Txt Ai Crawler Audit to Claude Code?
Run this command in your terminal: claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit
What are the best alternatives to Robots Txt Ai Crawler Audit?
The closest alternatives to Robots Txt Ai Crawler Audit are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages.
Where this comes from
- Description: from the official MCP Registry entry.
- Install steps: from the official MCP Registry entry.
Alternatives to Robots Txt Ai Crawler Audit

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.9K GitHub stars
- Web search and scraping

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping

OnlineJobs.ph Philippine remote and VA jobs with parsed wages, billed per job
- Remote
- Paid
- Web search and scraping

ImportYeti US import records and company profiles as JSON, billed per row
- Remote
- Paid
- Web search and scraping