Skip to content
MCP server

Commoncrawl

Common Crawl's public web archive — find every time a URL was crawled since 2008 and…

Install
Pricing
Open source
Runs
Locally or Remotely
Version
0.2.1
Listing updated
5 days ago

About Commoncrawl

Commoncrawl is an open-source MCP server. It can run locally on your machine or as a hosted remote server.

Common Crawl's public web archive — find every time a URL was crawled since 2008 and read the…

Install

Run in a terminal

claude mcp add --transport http commoncrawl https://gateway.pipeworx.io/commoncrawl/mcp

Add to Settings → Connectors → Add custom connector

https://gateway.pipeworx.io/commoncrawl/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "commoncrawl": {
      "url": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "commoncrawl": {
      "type": "http",
      "url": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "commoncrawl": {
      "serverUrl": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "commoncrawl": {
      "type": "http",
      "url": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}

Configuration

  • PIPEWORX_API_KEYOptional Pipeworx API key. Without it calls are anonymous and capped daily; a free key raises the cap.

Pricing

Open source

Prices change. Check pipeworx.io before you buy.

Commoncrawl FAQ

Is Commoncrawl free?

Yes. Commoncrawl is open source, so you can use and self-host it for free.

How do I add Commoncrawl to Claude Code?

Run this command in your terminal: claude mcp add --transport http commoncrawl https://gateway.pipeworx.io/commoncrawl/mcp