Skip to content
MCP server

AI Crawler Access Check by Citant.ai

Free check of whether GPTBot, ClaudeBot and PerplexityBot can really reach a website

Status and history

Answered our latest tool-list check 4 hours ago with 1 tools. It will be marked verified after 3 successful checks in 7 days.

Install
Pricing
Free
Platforms
API, Web
Runs
Remotely
Version
1.1.0
Listing updated
3 days ago

About AI Crawler Access Check by Citant.ai

AI Crawler Access Check by Citant.ai is a free MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Reading robots.txt only shows what a site asks of crawlers. This server fetches the target as each AI crawler would, so it can also catch blocks applied by a firewall or CDN. It runs the same 21 checks as the AI Crawler Access Report from Citant.ai.

The single tool, check_ai_crawler_access, takes a domain or URL and returns an AI Access Score from 0 to 100 with three sub-scores for live answers, training data and search index. Each crawler gets a verdict comparing what robots.txt says with what a real fetch received, plus up to eight critical or warning findings with a fix, and a link to the full report.

A scan takes 15 to 60 seconds and results for the same domain are cached for 24 hours. The server is remote, read-only and needs no login or API key, and it reads public pages only.

Install

Run in a terminal

claude mcp add --transport http ai-crawler-check 'https://check.citant.ai/mcp?src=registry'

Add to Settings → Connectors → Add custom connector

https://check.citant.ai/mcp?src=registry
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-check": {
      "url": "https://check.citant.ai/mcp?src=registry"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://check.citant.ai/mcp?src=registry"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "ai-crawler-check": {
      "serverUrl": "https://check.citant.ai/mcp?src=registry"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "ai-crawler-check": {
      "type": "http",
      "url": "https://check.citant.ai/mcp?src=registry"
    }
  }
}

Tools this server provides (1)

Read directly from the server.

check_ai_crawler_access
Checks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website. Goes beyond robots.txt: fetches the site as each crawler and detects firewall/CDN blocks (Cloudflare, Akamai, Imperva), challenge pages, noindex and llms.txt. Returns a 0-100 AI Access Score, a per-crawler verdict and the top issues with fixes. Takes

What it does

Real per-bot fetches
Requests the site as each AI crawler, not just reading robots.txt.
Three sub-scores
Separates live answers, training data and search index access.
Firewall and CDN detection
Finds blocks that robots.txt readers miss.
Fixes included
Up to eight findings, each with a suggested fix.
No credentials
Free, read-only, with no login or API key.
24-hour cache
Repeat scans of the same domain reuse results for a day.

Good for

  • Check whether ChatGPT can crawl your homepage
  • Find out if the firewall blocks GPTBot while robots.txt allows it
  • Get the top fix for a pricing page
  • Audit AI visibility before a content push

Pricing

Free

Prices change. Check check.citant.ai before you buy.

AI Crawler Access Check by Citant.ai FAQ

Does it need an API key?

No. It is free, read-only and needs no login or key.

How long does a scan take?

15 to 60 seconds, and the same domain is cached for 24 hours.

What do I give it?

A domain or URL, as the input to check_ai_crawler_access.

Will it change my site?

No. It reads public pages only.

Is AI Crawler Access Check by Citant.ai free?

Yes. AI Crawler Access Check by Citant.ai is free to use.

How do I add AI Crawler Access Check by Citant.ai to Claude Code?

Run this command in your terminal: claude mcp add --transport http ai-crawler-check 'https://check.citant.ai/mcp?src=registry'

What are the best alternatives to AI Crawler Access Check by Citant.ai?

The closest alternatives to AI Crawler Access Check by Citant.ai are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

Alternatives to AI Crawler Access Check by Citant.ai

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  4. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  5. Calven

    MCP server

    Product marketing context from Calven delivered into Claude, ChatGPT and other AI tools

    Not checked yet
    • Remote
    • Free trial
    • 3 GitHub stars
    • Marketing and SEO
  6. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping