Skip to content
MCP server

Robots Txt Ai Crawler Audit

Checks robots.txt line by line and says what each AI crawler token actually controls…

Runs locallyStatus and history
Install
Pricing
Open source
Runs
Locally
Version
0.1.3
Listing updated
5 days ago

Similar MCP servers

About Robots Txt Ai Crawler Audit

Robots Txt Ai Crawler Audit is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

Install

Run in a terminal

claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit

Add to claude_desktop_config.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "robots-txt-ai-crawler-audit": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}

Using several servers? Build one config file with the MCP config generator.

Pricing

Open source

Robots Txt Ai Crawler Audit FAQ

Is Robots Txt Ai Crawler Audit free?

Yes. Robots Txt Ai Crawler Audit is open source, so you can use and self-host it for free.

How do I add Robots Txt Ai Crawler Audit to Claude Code?

Run this command in your terminal: claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit

What are the best alternatives to Robots Txt Ai Crawler Audit?

The closest alternatives to Robots Txt Ai Crawler Audit are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages.

Where this comes from

  • Description: from the official MCP Registry entry.
  • Install steps: from the official MCP Registry entry.

How AIPoweredBot checks servers

Alternatives to Robots Txt Ai Crawler Audit

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.9K GitHub stars
    • Web search and scraping
  3. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  4. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping