# ShadowCrawl

> ShadowCrawl is an open-source MCP server for web search and scraping. Self-hosted MCP server for stealth web search, anti-bot scraping and browser automation. It is available on CLI, self-hosted, API, vscode, macOS, Linux and Windows.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, self-hosted, API, vscode, macOS, Linux, Windows
- **Repository:** https://github.com/devshero/shadowcrawl
- **MCP Registry name:** io.github.DevsHero/ShadowCrawl
- **Categories:** [web search and scraping](https://ai-powered.com/categories/web-search-and-scraping)
- **Listing updated:** 2026-11-18

## About

The README describes this as a single self-hostable Rust binary that gives AI agents search, page extraction and stateful browser automation over MCP (stdio) or an optional HTTP server. Fetching falls back in stages: plain retrieval, then Chromium CDP rendering, then a visible-browser human-in-the-loop step when a CAPTCHA or login wall blocks the request.

The browser side is its own engine. An agent sends an array of steps to one omni-tool, which covers navigation, locators, assertions, screenshots, route mocks, network capture and cookie or storage handling. A persistent isolated profile keeps sessions between calls, and a separate auth tool opens that profile in a visible window so a person can solve a challenge once.

The project is aimed at developers wiring research or scraping into VS Code, Cursor or Claude Desktop, and at teams who want search results and extracted text trimmed for LLM context. Its configuration is extensive, with timeouts, proxy lists, host-level throttling and optional LanceDB memory all set through environment variables.

## Features

- **Multi-engine web search:** Queries several search engines, merges and deduplicates results, and can return page content in the same call.
- **Staged anti-bot fallbacks:** Moves from native fetching to CDP rendering and block-aware retries before escalating to a human-assisted browser.
- **Stateful browser automation:** Runs Playwright-style action arrays in one omni-tool with a persistent profile that keeps cookies and session state.
- **Human-in-the-loop auth:** Opens the agent's profile in a visible window so a person can complete CAPTCHAs or OAuth logins once.
- **Structured extraction and research:** Extracts fields from pages and runs multi-hop deep research with optional LLM synthesis via OpenAI-compatible endpoints.
- **Research memory:** Stores past research in a LanceDB-backed local store that agents can search before fetching live pages.

## Good for

- Giving a coding agent live web search and page extraction inside an IDE
- Scraping protected listing or job pages that block plain HTTP requests
- Running UI or end-to-end test flows with route mocks and trace export through an agent
- Compiling multi-source research summaries with optional local or hosted LLM synthesis

## Tools

- `web_search`: URL discovery, optionally with page content in the same call.
- `web_fetch`: Unified fetch in single, batch or crawl mode.
- `extract_fields`: Structured extraction from a page.
- `scout_browser_automate`: Stateful browser automation via a steps array.
- `scout_agent_profile_auth`: Opens the agent profile in a visible window for CAPTCHA or login.
- `scout_browser_close`: Closes the browser session.
- `visual_scout`: Visual check for auth-gated pages.
- `hitl_web_fetch`: Human-assisted fetch for challenge or auth walls.
- `memory_search`: Searches LanceDB-backed research history.
- `deep_research`: Multi-hop search, scrape and synthesis.

## FAQ

### How is the server run?

The README says it is distributed as prebuilt binaries on GitHub Releases or built from source with Cargo, and that cortex-scout-mcp is the MCP stdio server while cortex-scout is an optional HTTP server.

### Does it need a proxy?

No. The README says the default is direct, no-proxy traffic, and proxy support is enabled only when IP_LIST_PATH and PROXY_SOURCE_PATH are set.

### Which MCP clients are covered in the setup guide?

The README gives configuration examples for VS Code, Cursor and Claude Desktop, with a multi-IDE guide in docs/IDE_SETUP.md.

### Is ShadowCrawl free?

Yes. ShadowCrawl is open source, so you can use and self-host it for free.

### What are the best alternatives to ShadowCrawl?

The closest alternatives to ShadowCrawl are acrawl, Scrapling MCP Server, Webshift, webclaw and Cat Burglar.

## Where this comes from

- Description: written from the project's README and its MCP Registry entry.

## Alternatives

- [acrawl](https://ai-powered.com/mcp/acrawl): acrawl is an open-source MCP server for browser automation.
- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Webshift](https://ai-powered.com/mcp/webshift-3): Webshift is an open-source MCP server for web search and scraping.
- [webclaw](https://ai-powered.com/mcp/webclaw): webclaw is a freemium MCP server for web search and scraping.
- [Cat Burglar](https://ai-powered.com/mcp/cat-burglar): Cat Burglar is a free MCP server for browser automation.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Stophy](https://ai-powered.com/mcp/stophy): Stophy is a freemium MCP server for web search and scraping.
- [Browserless](https://ai-powered.com/mcp/browserless): Browserless is a freemium MCP server for browser automation.

---
Source: https://ai-powered.com/mcp/shadowcrawl
