# Atalaya AI crawler check

> Atalaya AI crawler check is a free MCP server for web search and scraping. Check which AI crawlers a website allows in robots.txt, with the rule behind each verdict. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Free
- **Platforms:** API
- **Website:** https://atalaya-ai.com/mcp-server/
- **MCP Registry name:** com.atalaya-ai/ai-crawler-check
- **Categories:** web search and scraping, marketing and SEO
- **Listing updated:** 2026-11-15

## About

This server answers one question for a domain you name: can the AI crawlers read it? It fetches only public files, namely robots.txt, llms.txt and the home page, and reports a verdict for each of 20 crawlers including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, along with the exact rule that decided it. It also notes whether llms.txt and a sitemap exist and hints at the platform or CDN in use.

A second tool lists the crawler tokens it checks, grouped by vendor and purpose (training, search index, or fetch on a user's request), which helps when writing a robots.txt that keeps AI search but opts out of training.

The maker is open about limits: robots.txt only shows what a site asks for, CDN or firewall rules are invisible from outside, and results are cached for an hour. It does not test JavaScript rendering or structured data. The domains checked are not stored.

## Features

- **Per-crawler verdicts:** Allowed or blocked status for 20 AI crawlers with the deciding rule.
- **llms.txt and sitemap check:** Reports whether each file is present.
- **Crawler catalogue:** Lists tokens by vendor and purpose.
- **Read-only:** Fetches only robots.txt, llms.txt and the home page.
- **English or Spanish:** A lang option switches the answer language.
- **No account:** Free, no key needed.

## Good for

- Check if ChatGPT search can read a site before launch
- Draft a robots.txt that allows AI search but blocks training
- Audit a client's robots.txt for AI crawler rules
- Confirm a robots change took effect

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://atalaya-ai.com/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "ai-crawler-check": {
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "ai-crawler-check": {
      "serverUrl": "https://atalaya-ai.com/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://atalaya-ai.com/mcp"
    }
  }
}
```

## Tools

- `check_ai_crawler_access`: Returns robots.txt verdicts for AI crawlers, llms.txt and sitemap presence and a platform hint
- `list_ai_crawlers`: Lists the crawler tokens checked, by vendor and purpose

## FAQ

### Is it free?

Yes. The server is free with no account or key.

### What does it fetch?

Only /robots.txt, /llms.txt and the home page of the domain you name.

### Can it see firewall blocks?

No. CDN, firewall and bot-protection rules act before robots.txt and are not visible from outside.

### Does it keep my domains?

The maker says it counts checks per day but does not store the domains.

### Is Atalaya AI crawler check free?

Yes. Atalaya AI crawler check is free to use.

### How do I add Atalaya AI crawler check to Claude Code?

Run this command in your terminal: claude mcp add --transport http ai-crawler-check https://atalaya-ai.com/mcp

### What are the best alternatives to Atalaya AI crawler check?

The closest alternatives to Atalaya AI crawler check are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [Calven](https://ai-powered.com/mcp/calven): Calven is an MCP server for marketing and seo.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [MCP](https://ai-powered.com/mcp/mcp-mwhffe53): MCP is a freemium MCP server for marketing and seo.

---
Source: https://ai-powered.com/mcp/atalaya-ai-crawler-check
