AI Crawler Access Check by Citant.ai
Free check of whether GPTBot, ClaudeBot and PerplexityBot can really reach a website
Answered our latest tool-list check 4 hours ago with 1 tools. It will be marked verified after 3 successful checks in 7 days.
- Pricing
- Free
- Platforms
- API, Web
- Runs
- Remotely
- Version
- 1.1.0
- Listing updated
- 3 days ago
About AI Crawler Access Check by Citant.ai
AI Crawler Access Check by Citant.ai is a free MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.
Reading robots.txt only shows what a site asks of crawlers. This server fetches the target as each AI crawler would, so it can also catch blocks applied by a firewall or CDN. It runs the same 21 checks as the AI Crawler Access Report from Citant.ai.
The single tool, check_ai_crawler_access, takes a domain or URL and returns an AI Access Score from 0 to 100 with three sub-scores for live answers, training data and search index. Each crawler gets a verdict comparing what robots.txt says with what a real fetch received, plus up to eight critical or warning findings with a fix, and a link to the full report.
A scan takes 15 to 60 seconds and results for the same domain are cached for 24 hours. The server is remote, read-only and needs no login or API key, and it reads public pages only.
Install
Run in a terminal
claude mcp add --transport http ai-crawler-check 'https://check.citant.ai/mcp?src=registry'Add to Settings → Connectors → Add custom connector
https://check.citant.ai/mcp?src=registryAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"ai-crawler-check": {
"url": "https://check.citant.ai/mcp?src=registry"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"ai-crawler-check": {
"type": "http",
"url": "https://check.citant.ai/mcp?src=registry"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"ai-crawler-check": {
"serverUrl": "https://check.citant.ai/mcp?src=registry"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"ai-crawler-check": {
"type": "http",
"url": "https://check.citant.ai/mcp?src=registry"
}
}
}Tools this server provides (1)
Read directly from the server.
- check_ai_crawler_access
- Checks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website. Goes beyond robots.txt: fetches the site as each crawler and detects firewall/CDN blocks (Cloudflare, Akamai, Imperva), challenge pages, noindex and llms.txt. Returns a 0-100 AI Access Score, a per-crawler verdict and the top issues with fixes. Takes
What it does
- Real per-bot fetches
- Requests the site as each AI crawler, not just reading robots.txt.
- Three sub-scores
- Separates live answers, training data and search index access.
- Firewall and CDN detection
- Finds blocks that robots.txt readers miss.
- Fixes included
- Up to eight findings, each with a suggested fix.
- No credentials
- Free, read-only, with no login or API key.
- 24-hour cache
- Repeat scans of the same domain reuse results for a day.
Good for
- Check whether ChatGPT can crawl your homepage
- Find out if the firewall blocks GPTBot while robots.txt allows it
- Get the top fix for a pricing page
- Audit AI visibility before a content push
Pricing
Free
Prices change. Check check.citant.ai before you buy.
AI Crawler Access Check by Citant.ai FAQ
Does it need an API key?
No. It is free, read-only and needs no login or key.
How long does a scan take?
15 to 60 seconds, and the same domain is cached for 24 hours.
What do I give it?
A domain or URL, as the input to check_ai_crawler_access.
Will it change my site?
No. It reads public pages only.
Is AI Crawler Access Check by Citant.ai free?
Yes. AI Crawler Access Check by Citant.ai is free to use.
How do I add AI Crawler Access Check by Citant.ai to Claude Code?
Run this command in your terminal: claude mcp add --transport http ai-crawler-check 'https://check.citant.ai/mcp?src=registry'
What are the best alternatives to AI Crawler Access Check by Citant.ai?
The closest alternatives to AI Crawler Access Check by Citant.ai are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.
Alternatives to AI Crawler Access Check by Citant.ai

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Google Ads + Meta Ads + SEO/GEO MCP
MCP serverHosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat
- 8 tools
- Remote
- Freemium
- 8.1K GitHub stars
- Marketing and SEO

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

Calven
MCP serverProduct marketing context from Calven delivered into Claude, ChatGPT and other AI tools
Not checked yet- Remote
- Free trial
- 3 GitHub stars
- Marketing and SEO

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping