Skip to content
MCP server

Webcrawl Anything

Search, read, crawl and extract from the web locally, with no API key and sources cited

Install
Pricing
Open source
Platforms
CLI, macOS, Windows
Runs
Locally or Remotely
Version
1.4.8
Listing updated
today

Similar MCP servers

About Webcrawl Anything

Webcrawl Anything is an open-source MCP server for web search and scraping. It can run locally on your machine or as a hosted remote server.

By default the engine runs on your own machine through npx @webcrawl/mcp, with no account or key. If you set WEBCRAWL_API_KEY, the same tools run on the hosted webcrawl.app service instead, and WEBCRAWL_MODE chooses local, cloud or auto. The package exits with an error rather than silently calling the cloud if the local engine fails to load. A compact schema profile trims tool descriptions for clients with small context windows.

Beyond plain search it covers structured verticals such as shopping, flights, hotels, real estate, business, security, academic, tech and news. Pages are returned as clean Markdown, including JavaScript-heavy sites, and the parse tools read PDFs with OCR, Office files, YouTube videos and audio or video. Crawls are capped at 25 pages per call, and page monitoring with email or webhook alerts is cloud-only.

An npx webcrawl setup command registers the server and installs an agent skill for Claude Code, Cursor, Codex, Windsurf or Gemini CLI. MIT licensed.

Install

Run in a terminal

claude mcp add --transport http webcrawl https://webcrawl.app/mcp

Add to Settings → Connectors → Add custom connector

https://webcrawl.app/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "webcrawl": {
      "url": "https://webcrawl.app/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "webcrawl": {
      "type": "http",
      "url": "https://webcrawl.app/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "webcrawl": {
      "serverUrl": "https://webcrawl.app/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "webcrawl": {
      "type": "http",
      "url": "https://webcrawl.app/mcp"
    }
  }
}

Using several servers? Build one config file with the MCP config generator.

Configuration

  • WEBCRAWL_API_KEYOptional. Leave unset to run fully local. Set a webcrawl.app key to use the hosted cloud.
  • WEBCRAWL_MODEOptional: local, cloud or auto. Default auto runs local unless an API key is set.
  • WEBCRAWL_MCP_SCHEMA_PROFILESet to compact for smaller tool schemas in context-limited clients.

Tools this server provides (10)

Taken from the project documentation; the running server may differ.

web_search
Search the web or structured verticals.
web_read
Fetch a URL and return clean Markdown.
web_parse
Parse PDFs, Office files, YouTube and media into Markdown.
web_parse_status
Check the status of a parse job.
web_map
Discover all URLs on a domain.
web_crawl
Crawl up to 25 pages from a URL.
web_extract
Extract structured JSON from a URL.
web_rerank
Rerank documents by relevance to a query.
web_monitor
Watch pages on a schedule with alerts (cloud only).
search_categories
List supported search categories and filters.

What it does

Local by default
Runs on your machine with no API key or account.
Search verticals
Searches shopping, flights, hotels, real estate, business, security and more.
Page reading
Returns clean Markdown from JavaScript-heavy pages.
Document parsing
Handles PDFs with OCR, Office files, YouTube and audio or video.
Structured extraction
Extracts JSON from a page using a prompt or schema.
Optional cloud
Hosted mode adds scheduled monitoring with alerts.

Good for

  • Research a topic with cited sources from inside an editor
  • Map every URL on a domain before a crawl
  • Extract product data from a page into JSON
  • Convert a PDF or Word file to Markdown for an agent

Pricing

Open source

MIT licensed; optional hosted cloud with an API key

Webcrawl Anything FAQ

Do I need an API key?

No. Local mode is the default and needs no key or account.

What is cloud-only?

Page monitoring with trigger conditions and email or webhook alerts.

How many pages can it crawl?

Up to 25 pages starting from a URL per crawl call.

What licence is it?

MIT.

Is Webcrawl Anything free?

Yes. Webcrawl Anything is open source, so you can use and self-host it for free.

How do I add Webcrawl Anything to Claude Code?

Run this command in your terminal: claude mcp add --transport http webcrawl https://webcrawl.app/mcp

What are the best alternatives to Webcrawl Anything?

The closest alternatives to Webcrawl Anything are Scrapling MCP Server, Firecrawl MCP Server, Douyin_TikTok_Download_API, SearXNG Search and machine_internet.

Where this comes from

  • Description: written from the project's README.
  • Install steps: from the official MCP Registry entry.

How AIPoweredBot checks servers