Skip to content

Crawl Readiness vs XERJ

Crawl Readiness is a freemium MCP server for web search and scraping. XERJ is an open-source MCP server for knowledge and memory.

  • XERJ is open source, while Crawl Readiness is freemium.
  • Crawl Readiness is best for find out whether gptbot or claudebot is blocked on a site.
  • XERJ is best for search local code and docs from an agent.
What it is
Crawl Readiness is a freemium MCP server for web search and scraping.
XERJ is an open-source MCP server for knowledge and memory.
Best for
Find out whether GPTBot or ClaudeBot is blocked on a site
Search local code and docs from an agent
Kind
MCP server
MCP server
Pricing
Freemium, from $39 a month
Open source
Platforms
API, CLI, macOS, Windows, Linux
CLI, Linux, macOS, self-hosted, API
Runs
Locally
Locally
GitHub stars
19.2K
Last commit
Nov 2026
License
Apache-2.0
Open source
Good for
Find out whether GPTBot or ClaudeBot is blocked on a site; Generate llms.txt from a sitemap; Audit a robots.txt for orphan rules and wildcard traps
Search local code and docs from an agent; Give an agent persistent memory notes; Look up real API definitions

What Crawl Readiness does

  • Crawler readiness score

    Scores a site 0-100 against 60 AI crawlers with a prioritised fix list.

  • Signed permission receipts

    Answers whether a use is allowed and returns an Ed25519-signed receipt.

  • Content parity check

    Compares what Chrome, GPTBot, ClaudeBot and PerplexityBot each receive.

  • JSON-LD validation

    Audits structured data against 20+ common types.

  • File generators

    Writes llms.txt, robots.txt presets and Organization or Article JSON-LD.

  • Answer Analysis trend

    Reads whether AI search named your brand or competitors over time.

What XERJ does

  • Single binary

    Search, vectors, facts and memory in one Rust program.

  • Evidence-backed answers

    Answers come with the program and evidence used.

  • Runs locally

    Works offline on a CPU, on a laptop or server.

  • Elasticsearch-compatible API

    Exposes an API compatible with Elasticsearch.

  • Reference coding

    Index open-source repos so agents consult real definitions.

  • MCP integration

    Stdio MCP server connects to a running node.

Crawl Readiness vs XERJ: questions

Is Crawl Readiness or XERJ cheaper?

XERJ is the cheaper option: it is open source.

What is the difference between Crawl Readiness and XERJ?

Crawl Readiness is a freemium MCP server for web search and scraping. XERJ is an open-source MCP server for knowledge and memory. XERJ is open source, while Crawl Readiness is freemium.