# Robots Txt Ai Crawler Audit

> Robots Txt Ai Crawler Audit is an open-source MCP server for web search and scraping. Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Open source
- **Repository:** https://github.com/jmshinhwa/readystack-themes
- **MCP Registry name:** io.github.jmshinhwa/robots-txt-ai-crawler-audit
- **Categories:** web search and scraping
- **Listing updated:** 2026-11-13

## About

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

## Install

### Claude Code (Terminal)

```sh
claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "robots-txt-ai-crawler-audit": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "robots-txt-ai-crawler-audit": {
      "command": "npx",
      "args": [
        "-y",
        "@readystack/robots-txt-ai-crawler-audit"
      ]
    }
  }
}
```

## FAQ

### Is Robots Txt Ai Crawler Audit free?

Yes. Robots Txt Ai Crawler Audit is open source, so you can use and self-host it for free.

### How do I add Robots Txt Ai Crawler Audit to Claude Code?

Run this command in your terminal: claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit

### What are the best alternatives to Robots Txt Ai Crawler Audit?

The closest alternatives to Robots Txt Ai Crawler Audit are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages.

## Where this comes from

- Description: from the official MCP Registry entry.
- Install steps: from the official MCP Registry entry.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages](https://ai-powered.com/mcp/onlinejobs-ph-scraper-apify-philippine-remote-and-va-jobs-wi): OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages is a paid MCP server for web search and scraping.
- [ImportYeti Scraper (Apify) - US import records and bills of lading](https://ai-powered.com/mcp/importyeti-scraper-apify-us-import-records-and-bills-of-ladi): ImportYeti Scraper (Apify) - US import records and bills of lading is a paid MCP server for web search and scraping.
- [Gupy Jobs Scraper (Apify) - Brazil jobs with descriptions](https://ai-powered.com/mcp/gupy-jobs-scraper-apify-brazil-jobs-with-descriptions): Gupy Jobs Scraper (Apify) - Brazil jobs with descriptions is a freemium MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/robots-txt-ai-crawler-audit
