# Crawlerpeek

> Crawlerpeek is an open-source MCP server for marketing and seo. Test whether GPTBot, ClaudeBot and other crawlers may fetch a URL under robots.txt. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** Web, CLI
- **Website:** https://crawlerpeek.com
- **Repository:** https://github.com/crawlerpeek/crawlerpeek
- **MCP Registry name:** io.github.crawlerpeek/crawlerpeek
- **Categories:** marketing and SEO, developer tools
- **Listing updated:** 2026-11-14

## About

Crawlerpeek answers a narrow question: can a given bot reach this page? For each crawler it applies robots.txt the way RFC 9309 describes (user-agent groups, wildcards, the $ anchor, longest match wins) and then fetches the page with that bot's user-agent string, reporting status, final URL, content type, size, title and whether the response looks like a block or challenge page. The web tool checks 24 bots from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and others, and shows training, search and user-triggered bots separately.

The npm package adds a CLI that can fail a CI job with --fail-on, a lint command for robots.txt mistakes, a scan of every URL in a sitemap, a robots.txt generator that welcomes AI crawlers, and a verify command that checks a log IP against operators' published ranges or reverse DNS. The website also shows what a bot sees without JavaScript and offers a fix prompt for Claude Code, Cursor or Lovable.

The MCP server is read-only and the package needs Node 20 or newer with no runtime dependencies. The web checker says it stores nothing about the address you enter.

## Features

- **Per-bot robots.txt check:** Applies RFC 9309 rules to each crawler and URL path.
- **Live fetch as the bot:** Fetches the page with the bot's user agent and flags block pages.
- **robots.txt lint:** Finds mistakes in a robots.txt from a URL, file or stdin.
- **Sitemap scan:** Checks every sitemap URL against robots.txt for each bot.
- **Crawler IP verification:** Checks an IP against published ranges or reverse DNS.
- **CI-friendly CLI:** Exits non-zero and writes Markdown summaries for GitHub Actions.

## Good for

- Confirm GPTBot is not blocked on a key page
- Fail a deploy when a bot loses access
- Check whether a request in server logs is a genuine Googlebot
- Generate a robots.txt that welcomes AI crawlers

## Install

### Claude Code (Terminal)

```sh
claude mcp add crawlerpeek -- npx -y crawlerpeek
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "crawlerpeek": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

## FAQ

### Is it free?

The web checker is free with no account, and the npm package is open source.

### Does the check run JavaScript?

No. It shows the raw HTML a bot would get without JavaScript.

### Can it verify that a crawler IP is real?

Yes, with the verify command, using published IP ranges or reverse DNS with forward confirmation.

### What does it need?

Node 20 or newer; run it with npx crawlerpeek.

### Is Crawlerpeek free?

Yes. Crawlerpeek is open source, so you can use and self-host it for free.

### How do I add Crawlerpeek to Claude Code?

Run this command in your terminal: claude mcp add crawlerpeek -- npx -y crawlerpeek

### What are the best alternatives to Crawlerpeek?

The closest alternatives to Crawlerpeek are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.

## Alternatives

- [REA](https://ai-powered.com/mcp/rea): REA is an open-source MCP server for developer tools.
- [Codebase Memory](https://ai-powered.com/mcp/codebase-memory): Codebase Memory is an open-source MCP server for knowledge and memory.
- [GitHub](https://ai-powered.com/mcp/github-2): GitHub is an open-source MCP server for developer tools.
- [Serena MCP: the IDE for your agent](https://ai-powered.com/mcp/serena-mcp-the-ide-for-your-agent): Serena MCP: the IDE for your agent is an open-source MCP server for developer tools.
- [XERJ](https://ai-powered.com/mcp/xerj): XERJ is an open-source MCP server for knowledge and memory.
- [Ouroboros](https://ai-powered.com/mcp/ouroboros): Ouroboros is an open-source MCP server for developer tools.
- [Claude Code Ultimate Guide](https://ai-powered.com/mcp/claude-code-ultimate-guide): Claude Code Ultimate Guide is an open-source MCP server for ai models and llm tooling.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.

---
Source: https://ai-powered.com/mcp/crawlerpeek
