Skip to content
MCP server

Atalaya AI crawler check

Check which AI crawlers a website allows in robots.txt, with the rule behind each verdict

Install
Pricing
Free
Platforms
API
Runs
Remotely
Version
1.1.0
Listing updated
2 days ago

About Atalaya AI crawler check

Atalaya AI crawler check is a free MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

This server answers one question for a domain you name: can the AI crawlers read it? It fetches only public files, namely robots.txt, llms.txt and the home page, and reports a verdict for each of 20 crawlers including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, along with the exact rule that decided it. It also notes whether llms.txt and a sitemap exist and hints at the platform or CDN in use.

A second tool lists the crawler tokens it checks, grouped by vendor and purpose (training, search index, or fetch on a user's request), which helps when writing a robots.txt that keeps AI search but opts out of training.

The maker is open about limits: robots.txt only shows what a site asks for, CDN or firewall rules are invisible from outside, and results are cached for an hour. It does not test JavaScript rendering or structured data. The domains checked are not stored.

Install

Run in a terminal

claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcp

Add to Settings → Connectors → Add custom connector

https://atalaya-ai.com/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-check": {
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "ai-crawler-check": {
      "serverUrl": "https://atalaya-ai.com/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}

Tools this server provides (2)

Taken from the project documentation; the running server may differ.

check_ai_crawler_access
Returns robots.txt verdicts for AI crawlers, llms.txt and sitemap presence and a platform hint
list_ai_crawlers
Lists the crawler tokens checked, by vendor and purpose

What it does

Per-crawler verdicts
Allowed or blocked status for 20 AI crawlers with the deciding rule.
llms.txt and sitemap check
Reports whether each file is present.
Crawler catalogue
Lists tokens by vendor and purpose.
Read-only
Fetches only robots.txt, llms.txt and the home page.
English or Spanish
A lang option switches the answer language.
No account
Free, no key needed.

Good for

  • Check if ChatGPT search can read a site before launch
  • Draft a robots.txt that allows AI search but blocks training
  • Audit a client's robots.txt for AI crawler rules
  • Confirm a robots change took effect

Pricing

Free

Free server; the vendor sells separate paid reports and monthly plans

Prices change. Check atalaya-ai.com before you buy.

Atalaya AI crawler check FAQ

Is it free?

Yes. The server is free with no account or key.

What does it fetch?

Only /robots.txt, /llms.txt and the home page of the domain you name.

Can it see firewall blocks?

No. CDN, firewall and bot-protection rules act before robots.txt and are not visible from outside.

Does it keep my domains?

The maker says it counts checks per day but does not store the domains.

Is Atalaya AI crawler check free?

Yes. Atalaya AI crawler check is free to use.

How do I add Atalaya AI crawler check to Claude Code?

Run this command in your terminal: claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcp

What are the best alternatives to Atalaya AI crawler check?

The closest alternatives to Atalaya AI crawler check are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

Alternatives to Atalaya AI crawler check

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  4. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  5. Calven

    MCP server

    Product marketing context from Calven delivered into Claude, ChatGPT and other AI tools

    Not checked yet
    • Remote
    • Free trial
    • 3 GitHub stars
    • Marketing and SEO
  6. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping