
Atalaya AI crawler check
Check which AI crawlers a website allows in robots.txt, with the rule behind each verdict
- Pricing
- Free
- Platforms
- API
- Runs
- Remotely
- Version
- 1.1.0
- Listing updated
- 2 days ago
About Atalaya AI crawler check
Atalaya AI crawler check is a free MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.
This server answers one question for a domain you name: can the AI crawlers read it? It fetches only public files, namely robots.txt, llms.txt and the home page, and reports a verdict for each of 20 crawlers including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, along with the exact rule that decided it. It also notes whether llms.txt and a sitemap exist and hints at the platform or CDN in use.
A second tool lists the crawler tokens it checks, grouped by vendor and purpose (training, search index, or fetch on a user's request), which helps when writing a robots.txt that keeps AI search but opts out of training.
The maker is open about limits: robots.txt only shows what a site asks for, CDN or firewall rules are invisible from outside, and results are cached for an hour. It does not test JavaScript rendering or structured data. The domains checked are not stored.
Install
Run in a terminal
claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcpAdd to Settings → Connectors → Add custom connector
https://atalaya-ai.com/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"ai-crawler-check": {
"url": "https://atalaya-ai.com/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"ai-crawler-check": {
"type": "http",
"url": "https://atalaya-ai.com/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"ai-crawler-check": {
"serverUrl": "https://atalaya-ai.com/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"ai-crawler-check": {
"type": "http",
"url": "https://atalaya-ai.com/mcp"
}
}
}Tools this server provides (2)
Taken from the project documentation; the running server may differ.
- check_ai_crawler_access
- Returns robots.txt verdicts for AI crawlers, llms.txt and sitemap presence and a platform hint
- list_ai_crawlers
- Lists the crawler tokens checked, by vendor and purpose
What it does
- Per-crawler verdicts
- Allowed or blocked status for 20 AI crawlers with the deciding rule.
- llms.txt and sitemap check
- Reports whether each file is present.
- Crawler catalogue
- Lists tokens by vendor and purpose.
- Read-only
- Fetches only robots.txt, llms.txt and the home page.
- English or Spanish
- A lang option switches the answer language.
- No account
- Free, no key needed.
Good for
- Check if ChatGPT search can read a site before launch
- Draft a robots.txt that allows AI search but blocks training
- Audit a client's robots.txt for AI crawler rules
- Confirm a robots change took effect
Pricing
Free
Free server; the vendor sells separate paid reports and monthly plans
Prices change. Check atalaya-ai.com before you buy.
Atalaya AI crawler check FAQ
Is it free?
Yes. The server is free with no account or key.
What does it fetch?
Only /robots.txt, /llms.txt and the home page of the domain you name.
Can it see firewall blocks?
No. CDN, firewall and bot-protection rules act before robots.txt and are not visible from outside.
Does it keep my domains?
The maker says it counts checks per day but does not store the domains.
Is Atalaya AI crawler check free?
Yes. Atalaya AI crawler check is free to use.
How do I add Atalaya AI crawler check to Claude Code?
Run this command in your terminal: claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcp
What are the best alternatives to Atalaya AI crawler check?
The closest alternatives to Atalaya AI crawler check are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.
Alternatives to Atalaya AI crawler check

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Google Ads + Meta Ads + SEO/GEO MCP
MCP serverHosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat
- 8 tools
- Remote
- Freemium
- 8.1K GitHub stars
- Marketing and SEO

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

Calven
MCP serverProduct marketing context from Calven delivered into Claude, ChatGPT and other AI tools
Not checked yet- Remote
- Free trial
- 3 GitHub stars
- Marketing and SEO

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping