Crawl Readiness vs XERJ
Crawl Readiness is a freemium MCP server for web search and scraping. XERJ is an open-source MCP server for knowledge and memory.
- XERJ is open source, while Crawl Readiness is freemium.
- Crawl Readiness is best for find out whether gptbot or claudebot is blocked on a site.
- XERJ is best for search local code and docs from an agent.
- What it is
- Crawl Readiness is a freemium MCP server for web search and scraping.XERJ is an open-source MCP server for knowledge and memory.
- Best for
- Find out whether GPTBot or ClaudeBot is blocked on a siteSearch local code and docs from an agent
- Kind
- MCP serverMCP server
- Pricing
- Freemium, from $39 a monthOpen source
- Platforms
- API, CLI, macOS, Windows, LinuxCLI, Linux, macOS, self-hosted, API
- Runs
- LocallyLocally
- GitHub stars
- 19.2K
- Last commit
- Nov 2026
- License
- Apache-2.0
- Open source
- Good for
- Find out whether GPTBot or ClaudeBot is blocked on a site; Generate llms.txt from a sitemap; Audit a robots.txt for orphan rules and wildcard trapsSearch local code and docs from an agent; Give an agent persistent memory notes; Look up real API definitions
What Crawl Readiness does
Crawler readiness score
Scores a site 0-100 against 60 AI crawlers with a prioritised fix list.
Signed permission receipts
Answers whether a use is allowed and returns an Ed25519-signed receipt.
Content parity check
Compares what Chrome, GPTBot, ClaudeBot and PerplexityBot each receive.
JSON-LD validation
Audits structured data against 20+ common types.
File generators
Writes llms.txt, robots.txt presets and Organization or Article JSON-LD.
Answer Analysis trend
Reads whether AI search named your brand or competitors over time.
What XERJ does
Single binary
Search, vectors, facts and memory in one Rust program.
Evidence-backed answers
Answers come with the program and evidence used.
Runs locally
Works offline on a CPU, on a laptop or server.
Elasticsearch-compatible API
Exposes an API compatible with Elasticsearch.
Reference coding
Index open-source repos so agents consult real definitions.
MCP integration
Stdio MCP server connects to a running node.
Crawl Readiness vs XERJ: questions
Is Crawl Readiness or XERJ cheaper?
XERJ is the cheaper option: it is open source.
What is the difference between Crawl Readiness and XERJ?
Crawl Readiness is a freemium MCP server for web search and scraping. XERJ is an open-source MCP server for knowledge and memory. XERJ is open source, while Crawl Readiness is freemium.
