Crawlerpeek
Test whether GPTBot, ClaudeBot and other crawlers may fetch a URL under robots.txt
- Pricing
- Open source
- Platforms
- Web, CLI
- Runs
- Locally
- Version
- 0.3.1
- Listing updated
- 2 days ago
About Crawlerpeek
Crawlerpeek is an open-source MCP server for marketing and seo. It runs locally on your machine and is started by your AI client.
Crawlerpeek answers a narrow question: can a given bot reach this page? For each crawler it applies robots.txt the way RFC 9309 describes (user-agent groups, wildcards, the $ anchor, longest match wins) and then fetches the page with that bot's user-agent string, reporting status, final URL, content type, size, title and whether the response looks like a block or challenge page. The web tool checks 24 bots from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and others, and shows training, search and user-triggered bots separately.
The npm package adds a CLI that can fail a CI job with --fail-on, a lint command for robots.txt mistakes, a scan of every URL in a sitemap, a robots.txt generator that welcomes AI crawlers, and a verify command that checks a log IP against operators' published ranges or reverse DNS. The website also shows what a bot sees without JavaScript and offers a fix prompt for Claude Code, Cursor or Lovable.
The MCP server is read-only and the package needs Node 20 or newer with no runtime dependencies. The web checker says it stores nothing about the address you enter.
Install
Run in a terminal
claude mcp add crawlerpeek -- npx -y crawlerpeekAdd to claude_desktop_config.json
{
"mcpServers": {
"crawlerpeek": {
"command": "npx",
"args": [
"-y",
"crawlerpeek"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"crawlerpeek": {
"command": "npx",
"args": [
"-y",
"crawlerpeek"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"crawlerpeek": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"crawlerpeek"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"crawlerpeek": {
"command": "npx",
"args": [
"-y",
"crawlerpeek"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"crawlerpeek": {
"command": "npx",
"args": [
"-y",
"crawlerpeek"
]
}
}
}What it does
- Per-bot robots.txt check
- Applies RFC 9309 rules to each crawler and URL path.
- Live fetch as the bot
- Fetches the page with the bot's user agent and flags block pages.
- robots.txt lint
- Finds mistakes in a robots.txt from a URL, file or stdin.
- Sitemap scan
- Checks every sitemap URL against robots.txt for each bot.
- Crawler IP verification
- Checks an IP against published ranges or reverse DNS.
- CI-friendly CLI
- Exits non-zero and writes Markdown summaries for GitHub Actions.
Good for
- Confirm GPTBot is not blocked on a key page
- Fail a deploy when a bot loses access
- Check whether a request in server logs is a genuine Googlebot
- Generate a robots.txt that welcomes AI crawlers
Pricing
Open source
Prices change. Check crawlerpeek.com before you buy.
Crawlerpeek FAQ
Is it free?
The web checker is free with no account, and the npm package is open source.
Does the check run JavaScript?
No. It shows the raw HTML a bot would get without JavaScript.
Can it verify that a crawler IP is real?
Yes, with the verify command, using published IP ranges or reverse DNS with forward confirmation.
What does it need?
Node 20 or newer; run it with npx crawlerpeek.
Is Crawlerpeek free?
Yes. Crawlerpeek is open source, so you can use and self-host it for free.
How do I add Crawlerpeek to Claude Code?
Run this command in your terminal: claude mcp add crawlerpeek -- npx -y crawlerpeek
What are the best alternatives to Crawlerpeek?
The closest alternatives to Crawlerpeek are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.
Alternatives to Crawlerpeek

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Codebase Memory
MCP serverLocal MCP server that builds a code knowledge graph for AI coding agents
Runs locally- Open source
- 48.7K GitHub stars
- Knowledge and memory

GitHub
MCP serverOfficial MCP server connecting AI assistants to GitHub repos, issues and workflows
Not checked yet- Remote
- Open source
- 34.5K GitHub stars
- Developer tools

Serena MCP: the IDE for your agent
MCP serverMCP toolkit giving coding agents symbol-level code retrieval and editing
Runs locally- Open source
- 31.5K GitHub stars
- Developer tools

XERJ
MCP serverLocal search engine for AI agents with cited answers over your code and docs
Runs locally- Open source
- 19.2K GitHub stars
- Knowledge and memory

Ouroboros
MCP serverSpec-first layer that pins requirements and verifies results of AI coding agent work
Runs locally- Open source
- 6.4K GitHub stars
- Developer tools