# Webcrawl Anything

> Webcrawl Anything is an open-source MCP server for web search and scraping. Search, read, crawl and extract from the web locally, with no API key and sources cited. It can run locally on your machine or as a hosted remote server.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, macOS, Windows, Linux, API
- **Repository:** https://github.com/brainfish-ai/webcrawl-anything/tree/HEAD/packages/mcp
- **MCP Registry name:** io.github.brainfish-ai/webcrawl
- **Categories:** [web search and scraping](https://ai-powered.com/categories/web-search-and-scraping)
- **Listing updated:** 2026-11-18

## About

By default the engine runs on your own machine through npx @webcrawl/mcp, with no account or key. If you set WEBCRAWL_API_KEY, the same tools run on the hosted webcrawl.app service instead, and WEBCRAWL_MODE chooses local, cloud or auto. The package exits with an error rather than silently calling the cloud if the local engine fails to load. A compact schema profile trims tool descriptions for clients with small context windows.

Beyond plain search it covers structured verticals such as shopping, flights, hotels, real estate, business, security, academic, tech and news. Pages are returned as clean Markdown, including JavaScript-heavy sites, and the parse tools read PDFs with OCR, Office files, YouTube videos and audio or video. Crawls are capped at 25 pages per call, and page monitoring with email or webhook alerts is cloud-only.

An npx webcrawl setup command registers the server and installs an agent skill for Claude Code, Cursor, Codex, Windsurf or Gemini CLI. MIT licensed.

## Features

- **Local by default:** Runs on your machine with no API key or account.
- **Search verticals:** Searches shopping, flights, hotels, real estate, business, security and more.
- **Page reading:** Returns clean Markdown from JavaScript-heavy pages.
- **Document parsing:** Handles PDFs with OCR, Office files, YouTube and audio or video.
- **Structured extraction:** Extracts JSON from a page using a prompt or schema.
- **Optional cloud:** Hosted mode adds scheduled monitoring with alerts.

## Good for

- Research a topic with cited sources from inside an editor
- Map every URL on a domain before a crawl
- Extract product data from a page into JSON
- Convert a PDF or Word file to Markdown for an agent

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http webcrawl https://webcrawl.app/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://webcrawl.app/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "webcrawl": {
      "url": "https://webcrawl.app/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "webcrawl": {
      "type": "http",
      "url": "https://webcrawl.app/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "webcrawl": {
      "serverUrl": "https://webcrawl.app/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "webcrawl": {
      "type": "http",
      "url": "https://webcrawl.app/mcp"
    }
  }
}
```

## Tools

- `web_search`: Search the web or structured verticals.
- `web_read`: Fetch a URL and return clean Markdown.
- `web_parse`: Parse PDFs, Office files, YouTube and media into Markdown.
- `web_parse_status`: Check the status of a parse job.
- `web_map`: Discover all URLs on a domain.
- `web_crawl`: Crawl up to 25 pages from a URL.
- `web_extract`: Extract structured JSON from a URL.
- `web_rerank`: Rerank documents by relevance to a query.
- `web_monitor`: Watch pages on a schedule with alerts (cloud only).
- `search_categories`: List supported search categories and filters.

## FAQ

### Do I need an API key?

No. Local mode is the default and needs no key or account.

### What is cloud-only?

Page monitoring with trigger conditions and email or webhook alerts.

### How many pages can it crawl?

Up to 25 pages starting from a URL per crawl call.

### What licence is it?

MIT.

### Is Webcrawl Anything free?

Yes. Webcrawl Anything is open source, so you can use and self-host it for free.

### How do I add Webcrawl Anything to Claude Code?

Run this command in your terminal: claude mcp add --transport http webcrawl https://webcrawl.app/mcp

### What are the best alternatives to Webcrawl Anything?

The closest alternatives to Webcrawl Anything are Scrapling MCP Server, Firecrawl MCP Server, Douyin_TikTok_Download_API, SearXNG Search and machine_internet.

## Where this comes from

- Description: written from the project's README.
- Install steps: from the official MCP Registry entry.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Douyin_TikTok_Download_API](https://ai-powered.com/mcp/douyin-tiktok-download-api): Douyin_TikTok_Download_API is an open-source MCP server for web search and scraping.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [machine_internet](https://ai-powered.com/mcp/machine-internet): machine_internet is an open-source MCP server for web search and scraping.
- [acrawl](https://ai-powered.com/mcp/acrawl): acrawl is an open-source MCP server for browser automation.
- [Countries](https://ai-powered.com/mcp/countries): Countries is a freemium MCP server for web search and scraping.
- [Listen Notes](https://ai-powered.com/mcp/listen-notes): Listen Notes is a freemium MCP server for knowledge and memory.

---
Source: https://ai-powered.com/mcp/webcrawl-anything
