Skip to content
MCP server

CrawlWitness

Pay-per-call check of what a domain permits AI crawlers to do, with signed receipts

Install
Pricing
Paid
Platforms
API
Runs
Remotely
Version
1.0.52
Listing updated
2 days ago

About CrawlWitness

CrawlWitness is a paid MCP server for legal and government. It runs as a hosted remote server, so there is nothing to install locally.

Before training on, retrieving from or indexing a site, a crawler can ask CrawlWitness whether that is allowed. The answer is a yes or no for a given domain, AI bot and intended use (train, RAG or search), derived from robots.txt, IETF AIPREF, W3C TDMRep, noai signals and RSL licences.

What sets it apart is proof. Each verdict can come with a signed receipt and an attestation anchored on Base, so you can later show what a site declared on a given date. The site also sells evidence files aimed at disputes and negotiations.

The MCP tools mirror the REST endpoints and are billed in USDC through the x402 protocol on Base or Solana, with no account needed for pay-per-call use. Monthly plans can also be paid in USDC or by card, and a seven-day trial key is offered. Integrations are listed for Scrapy and Crawlee with Playwright.

Install

Run in a terminal

claude mcp add --transport http oracle https://api.crawlwitness.com/mcp

Add to Settings → Connectors → Add custom connector

https://api.crawlwitness.com/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "oracle": {
      "url": "https://api.crawlwitness.com/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "oracle": {
      "type": "http",
      "url": "https://api.crawlwitness.com/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "oracle": {
      "serverUrl": "https://api.crawlwitness.com/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "oracle": {
      "type": "http",
      "url": "https://api.crawlwitness.com/mcp"
    }
  }
}

Tools this server provides (3)

Taken from the project documentation; the running server may differ.

can_ai_use
Returns a yes/no verdict for a domain, AI bot and use.
check_ai_consent
Returns the full AI consent report for a domain.
check_wallet
Checks a wallet against OFAC sanctions.

What it does

Yes/no verdict
Answers whether a given bot may use a domain for a chosen purpose.
Full consent report
Returns the detailed signals behind a verdict.
Wallet sanctions check
Screens a wallet address against OFAC lists.
Batch and manifest calls
Checks up to 100 domains or wallets, or up to 1000 URLs, per request.
Signed receipts
Verdicts can be anchored on Base for later proof.
x402 payments
Agents pay per call in USDC with no sign-up.

Good for

  • Gate a Scrapy or Crawlee crawler on consent signals
  • Record what a site declared before using its pages in a RAG index
  • Screen a counterparty wallet before a payment
  • Keep evidence for a licensing dispute

Pricing

Paid

Pay per call from $0.001 in USDC (x402); monthly plans and a 7-day trial key

Prices change. Check api.crawlwitness.com before you buy.

CrawlWitness FAQ

How much does a verdict cost?

The pricing page lists $0.001 for a yes/no verdict and $0.005 for a full consent report.

Is there a free trial?

Yes, a 7-day trial key with no wallet or card.

Which signals does it read?

robots.txt, IETF AIPREF, W3C TDMRep, noai and RSL licences.

Which MCP tools exist?

can_ai_use, check_ai_consent and check_wallet are named.

Is CrawlWitness free?

No. CrawlWitness is a paid product.

How do I add CrawlWitness to Claude Code?

Run this command in your terminal: claude mcp add --transport http oracle https://api.crawlwitness.com/mcp

What are the best alternatives to CrawlWitness?

The closest alternatives to CrawlWitness are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.

Alternatives to CrawlWitness

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  4. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  5. Search AI tools, agents and MCP servers and get install configs from inside your AI assistant.

    • 4 tools
    • Remote
    • Free
    • Developer tools
  6. New York LLC publication rules, deadline, county prices, clerks and notice text

    • 8 tools
    • Remote
    • Free
    • Legal and government