Crawlerpeek vs XERJ
Crawlerpeek is an open-source MCP server for marketing and seo. XERJ is an open-source MCP server for knowledge and memory.
- Crawlerpeek is best for confirm gptbot is not blocked on a key page.
- XERJ is best for search local code and docs from an agent.
- What it is
- Crawlerpeek is an open-source MCP server for marketing and seo.XERJ is an open-source MCP server for knowledge and memory.
- Best for
- Confirm GPTBot is not blocked on a key pageSearch local code and docs from an agent
- Kind
- MCP serverMCP server
- Pricing
- Open sourceOpen source
- Platforms
- Web, CLICLI, Linux, macOS, self-hosted, API
- Runs
- LocallyLocally
- GitHub stars
- 19.2K
- Last commit
- Nov 2026
- License
- Apache-2.0
- Open source
- Good for
- Confirm GPTBot is not blocked on a key page; Fail a deploy when a bot loses access; Check whether a request in server logs is a genuine GooglebotSearch local code and docs from an agent; Give an agent persistent memory notes; Look up real API definitions
What Crawlerpeek does
Per-bot robots.txt check
Applies RFC 9309 rules to each crawler and URL path.
Live fetch as the bot
Fetches the page with the bot's user agent and flags block pages.
robots.txt lint
Finds mistakes in a robots.txt from a URL, file or stdin.
Sitemap scan
Checks every sitemap URL against robots.txt for each bot.
Crawler IP verification
Checks an IP against published ranges or reverse DNS.
CI-friendly CLI
Exits non-zero and writes Markdown summaries for GitHub Actions.
What XERJ does
Single binary
Search, vectors, facts and memory in one Rust program.
Evidence-backed answers
Answers come with the program and evidence used.
Runs locally
Works offline on a CPU, on a laptop or server.
Elasticsearch-compatible API
Exposes an API compatible with Elasticsearch.
Reference coding
Index open-source repos so agents consult real definitions.
MCP integration
Stdio MCP server connects to a running node.
Crawlerpeek vs XERJ: questions
Is Crawlerpeek or XERJ cheaper?
Both are open source. Compare current plans on each official site.
What is the difference between Crawlerpeek and XERJ?
Crawlerpeek is an open-source MCP server for marketing and seo. XERJ is an open-source MCP server for knowledge and memory.
