Skip to content
MCP server

Botcard

Cited registry of AI crawler robots.txt tokens and a robots.txt checker

Install
Pricing
Free
Platforms
API, Web
Runs
Remotely
Version
2026-10-18.18
Listing updated
1 day ago

About Botcard

Botcard is a free MCP server for marketing and seo. It runs as a hosted remote server, so there is nothing to install locally.

Botcard tells training, search, person-triggered fetching and open crawling apart for AI crawlers. Its registry holds 125 cards from 24 vendors, each citing the vendor's own page and noting when it was checked, with a field such as whether a disallow is an opt-out, whether robots.txt may not apply, or whether the token is only a control.

Four read-only tools are exposed. A checker takes pasted robots.txt text and a path and returns allow or disallow for each token under RFC 9309, along with the vendor's stance. Other tools list crawlers, fetch one card, and suggest a robots.txt from a policy such as allow search and block training. The service never fetches the host you ask about, and the response reports fetched_target false.

The cards flag details that matter, for example that Google-Extended has no user agent of its own and does not change Search inclusion, that OAI-SearchBot is independent of GPTBot, and that Perplexity-User and Meta-ExternalFetcher may ignore robots.txt. No key is needed.

Install

Run in a terminal

claude mcp add --transport http botcard https://whosthatbot.com/mcp

Add to Settings → Connectors → Add custom connector

https://whosthatbot.com/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "botcard": {
      "url": "https://whosthatbot.com/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "botcard": {
      "type": "http",
      "url": "https://whosthatbot.com/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "botcard": {
      "serverUrl": "https://whosthatbot.com/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "botcard": {
      "type": "http",
      "url": "https://whosthatbot.com/mcp"
    }
  }
}

Tools this server provides (4)

Taken from the project documentation; the running server may differ.

check_robots
Checks pasted robots.txt against crawler tokens for a path.
list_crawlers
Lists crawlers in the registry.
get_crawler
Returns one crawler's card.
suggest_robots
Suggests a robots.txt from a stated policy.

What it does

Cited crawler registry
125 cards across 24 vendors with source citations and check dates.
robots.txt checker
Answers allow or disallow per token for a path under RFC 9309.
Policy-based generator
Suggests a robots.txt from rules such as allow search, block training.
Purpose classification
Separates training, search, user fetch and open crawls.
Control tokens handled
Notes tokens like Google-Extended that do not crawl.
No fetching
Works only on pasted text and never visits a site.

Good for

  • Block AI training crawlers but allow search
  • Check whether a robots.txt blocks GPTBot
  • Look up what a specific crawler token does
  • Generate a robots.txt that follows a stated policy

Pricing

Free

Free, no key

Prices change. Check whosthatbot.com before you buy.

Botcard FAQ

Does it visit my site?

No, it only reads the robots.txt text you provide.

How many crawlers are covered?

125 cards from 24 vendors.

Is a key needed?

No.

Does blocking Google-Extended remove me from Search?

The Botcard entry says it does not change Search inclusion or ranking.

Is Botcard free?

Yes. Botcard is free to use.

How do I add Botcard to Claude Code?

Run this command in your terminal: claude mcp add --transport http botcard https://whosthatbot.com/mcp

What are the best alternatives to Botcard?

The closest alternatives to Botcard are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.

Alternatives to Botcard

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  3. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  4. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  5. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory
  6. Ouroboros

    MCP server

    Spec-first layer that pins requirements and verifies results of AI coding agent work

    Runs locally
    • Open source
    • 6.4K GitHub stars
    • Developer tools