Commoncrawl
Common Crawl's public web archive — find every time a URL was crawled since 2008 and…
- Pricing
- Open source
- Runs
- Locally or Remotely
- Version
- 0.2.1
- Listing updated
- 5 days ago
About Commoncrawl
Commoncrawl is an open-source MCP server. It can run locally on your machine or as a hosted remote server.
Common Crawl's public web archive — find every time a URL was crawled since 2008 and read the…
Install
Run in a terminal
claude mcp add --transport http commoncrawl https://gateway.pipeworx.io/commoncrawl/mcpAdd to Settings → Connectors → Add custom connector
https://gateway.pipeworx.io/commoncrawl/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"commoncrawl": {
"url": "https://gateway.pipeworx.io/commoncrawl/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"commoncrawl": {
"type": "http",
"url": "https://gateway.pipeworx.io/commoncrawl/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"commoncrawl": {
"serverUrl": "https://gateway.pipeworx.io/commoncrawl/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"commoncrawl": {
"type": "http",
"url": "https://gateway.pipeworx.io/commoncrawl/mcp"
}
}
}Configuration
PIPEWORX_API_KEYOptional Pipeworx API key. Without it calls are anonymous and capped daily; a free key raises the cap.
Pricing
Open source
Prices change. Check pipeworx.io before you buy.
Commoncrawl FAQ
Is Commoncrawl free?
Yes. Commoncrawl is open source, so you can use and self-host it for free.
How do I add Commoncrawl to Claude Code?
Run this command in your terminal: claude mcp add --transport http commoncrawl https://gateway.pipeworx.io/commoncrawl/mcp