# AI Crawler Access Check by Citant.ai

> AI Crawler Access Check by Citant.ai is a free MCP server for web search and scraping. Free check of whether GPTBot, ClaudeBot and PerplexityBot can really reach a website. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Free
- **Platforms:** API, Web
- **Website:** https://check.citant.ai/mcp-server
- **MCP Registry name:** io.github.hamzawamiqhq/ai-crawler-check
- **Categories:** web search and scraping, marketing and SEO
- **Listing updated:** 2026-11-15

## About

Reading robots.txt only shows what a site asks of crawlers. This server fetches the target as each AI crawler would, so it can also catch blocks applied by a firewall or CDN. It runs the same 21 checks as the AI Crawler Access Report from Citant.ai.

The single tool, check_ai_crawler_access, takes a domain or URL and returns an AI Access Score from 0 to 100 with three sub-scores for live answers, training data and search index. Each crawler gets a verdict comparing what robots.txt says with what a real fetch received, plus up to eight critical or warning findings with a fix, and a link to the full report.

A scan takes 15 to 60 seconds and results for the same domain are cached for 24 hours. The server is remote, read-only and needs no login or API key, and it reads public pages only.

## Features

- **Real per-bot fetches:** Requests the site as each AI crawler, not just reading robots.txt.
- **Three sub-scores:** Separates live answers, training data and search index access.
- **Firewall and CDN detection:** Finds blocks that robots.txt readers miss.
- **Fixes included:** Up to eight findings, each with a suggested fix.
- **No credentials:** Free, read-only, with no login or API key.
- **24-hour cache:** Repeat scans of the same domain reuse results for a day.

## Good for

- Check whether ChatGPT can crawl your homepage
- Find out if the firewall blocks GPTBot while robots.txt allows it
- Get the top fix for a pricing page
- Audit AI visibility before a content push

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http ai-crawler-check 'https://check.citant.ai/mcp?src=registry'
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://check.citant.ai/mcp?src=registry
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "ai-crawler-check": {
      "url": "https://check.citant.ai/mcp?src=registry"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://check.citant.ai/mcp?src=registry"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "ai-crawler-check": {
      "serverUrl": "https://check.citant.ai/mcp?src=registry"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://check.citant.ai/mcp?src=registry"
    }
  }
}
```

## Tools

- `check_ai_crawler_access`: Checks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website. Goes beyond robots.txt: fetches the site as each crawler and detects firewall/CDN blocks (Cloudflare, Akamai, Imperva), challenge pages, noindex and llms.txt. Returns a 0-100 AI Access Score, a per-crawler verdict and the top issues with fixes. Takes 

## FAQ

### Does it need an API key?

No. It is free, read-only and needs no login or key.

### How long does a scan take?

15 to 60 seconds, and the same domain is cached for 24 hours.

### What do I give it?

A domain or URL, as the input to check_ai_crawler_access.

### Will it change my site?

No. It reads public pages only.

### Is AI Crawler Access Check by Citant.ai free?

Yes. AI Crawler Access Check by Citant.ai is free to use.

### How do I add AI Crawler Access Check by Citant.ai to Claude Code?

Run this command in your terminal: claude mcp add --transport http ai-crawler-check 'https://check.citant.ai/mcp?src=registry'

### What are the best alternatives to AI Crawler Access Check by Citant.ai?

The closest alternatives to AI Crawler Access Check by Citant.ai are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [Calven](https://ai-powered.com/mcp/calven): Calven is an MCP server for marketing and seo.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [MCP](https://ai-powered.com/mcp/mcp-mwhffe53): MCP is a freemium MCP server for marketing and seo.

---
Source: https://ai-powered.com/mcp/ai-crawler-access-check-by-citant-ai
