Skip to content
MCP server

Crawlerpeek

Test whether GPTBot, ClaudeBot and other crawlers may fetch a URL under robots.txt

Runs locallyStatus and history
Install
Pricing
Open source
Platforms
Web, CLI
Runs
Locally
Version
0.3.1
Listing updated
2 days ago

About Crawlerpeek

Crawlerpeek is an open-source MCP server for marketing and seo. It runs locally on your machine and is started by your AI client.

Crawlerpeek answers a narrow question: can a given bot reach this page? For each crawler it applies robots.txt the way RFC 9309 describes (user-agent groups, wildcards, the $ anchor, longest match wins) and then fetches the page with that bot's user-agent string, reporting status, final URL, content type, size, title and whether the response looks like a block or challenge page. The web tool checks 24 bots from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and others, and shows training, search and user-triggered bots separately.

The npm package adds a CLI that can fail a CI job with --fail-on, a lint command for robots.txt mistakes, a scan of every URL in a sitemap, a robots.txt generator that welcomes AI crawlers, and a verify command that checks a log IP against operators' published ranges or reverse DNS. The website also shows what a bot sees without JavaScript and offers a fix prompt for Claude Code, Cursor or Lovable.

The MCP server is read-only and the package needs Node 20 or newer with no runtime dependencies. The web checker says it stores nothing about the address you enter.

Install

Run in a terminal

claude mcp add crawlerpeek -- npx -y crawlerpeek

Add to claude_desktop_config.json

{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "crawlerpeek": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}

What it does

Per-bot robots.txt check
Applies RFC 9309 rules to each crawler and URL path.
Live fetch as the bot
Fetches the page with the bot's user agent and flags block pages.
robots.txt lint
Finds mistakes in a robots.txt from a URL, file or stdin.
Sitemap scan
Checks every sitemap URL against robots.txt for each bot.
Crawler IP verification
Checks an IP against published ranges or reverse DNS.
CI-friendly CLI
Exits non-zero and writes Markdown summaries for GitHub Actions.

Good for

  • Confirm GPTBot is not blocked on a key page
  • Fail a deploy when a bot loses access
  • Check whether a request in server logs is a genuine Googlebot
  • Generate a robots.txt that welcomes AI crawlers

Pricing

Open source

Prices change. Check crawlerpeek.com before you buy.

Crawlerpeek FAQ

Is it free?

The web checker is free with no account, and the npm package is open source.

Does the check run JavaScript?

No. It shows the raw HTML a bot would get without JavaScript.

Can it verify that a crawler IP is real?

Yes, with the verify command, using published IP ranges or reverse DNS with forward confirmation.

What does it need?

Node 20 or newer; run it with npx crawlerpeek.

Is Crawlerpeek free?

Yes. Crawlerpeek is open source, so you can use and self-host it for free.

How do I add Crawlerpeek to Claude Code?

Run this command in your terminal: claude mcp add crawlerpeek -- npx -y crawlerpeek

What are the best alternatives to Crawlerpeek?

The closest alternatives to Crawlerpeek are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.

Alternatives to Crawlerpeek

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  3. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  4. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  5. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory
  6. Ouroboros

    MCP server

    Spec-first layer that pins requirements and verifies results of AI coding agent work

    Runs locally
    • Open source
    • 6.4K GitHub stars
    • Developer tools