Skip to content
MCP server

ShadowCrawl

Self-hosted MCP server for stealth web search, anti-bot scraping and browser automation

Pricing
Open source
Platforms
CLI, self-hosted, API
Version
2.3.0
Listing updated
today

Similar MCP servers

About ShadowCrawl

ShadowCrawl is an open-source MCP server for web search and scraping. It is available on CLI, self-hosted, API, vscode, macOS, Linux and Windows.

The README describes this as a single self-hostable Rust binary that gives AI agents search, page extraction and stateful browser automation over MCP (stdio) or an optional HTTP server. Fetching falls back in stages: plain retrieval, then Chromium CDP rendering, then a visible-browser human-in-the-loop step when a CAPTCHA or login wall blocks the request.

The browser side is its own engine. An agent sends an array of steps to one omni-tool, which covers navigation, locators, assertions, screenshots, route mocks, network capture and cookie or storage handling. A persistent isolated profile keeps sessions between calls, and a separate auth tool opens that profile in a visible window so a person can solve a challenge once.

The project is aimed at developers wiring research or scraping into VS Code, Cursor or Claude Desktop, and at teams who want search results and extracted text trimmed for LLM context. Its configuration is extensive, with timeouts, proxy lists, host-level throttling and optional LanceDB memory all set through environment variables.

Tools this server provides (10)

Taken from the project documentation; the running server may differ.

web_search
URL discovery, optionally with page content in the same call.
web_fetch
Unified fetch in single, batch or crawl mode.
extract_fields
Structured extraction from a page.
scout_browser_automate
Stateful browser automation via a steps array.
scout_agent_profile_auth
Opens the agent profile in a visible window for CAPTCHA or login.
scout_browser_close
Closes the browser session.
visual_scout
Visual check for auth-gated pages.
hitl_web_fetch
Human-assisted fetch for challenge or auth walls.
memory_search
Searches LanceDB-backed research history.
deep_research
Multi-hop search, scrape and synthesis.

What it does

Multi-engine web search
Queries several search engines, merges and deduplicates results, and can return page content in the same call.
Staged anti-bot fallbacks
Moves from native fetching to CDP rendering and block-aware retries before escalating to a human-assisted browser.
Stateful browser automation
Runs Playwright-style action arrays in one omni-tool with a persistent profile that keeps cookies and session state.
Human-in-the-loop auth
Opens the agent's profile in a visible window so a person can complete CAPTCHAs or OAuth logins once.
Structured extraction and research
Extracts fields from pages and runs multi-hop deep research with optional LLM synthesis via OpenAI-compatible endpoints.
Research memory
Stores past research in a LanceDB-backed local store that agents can search before fetching live pages.

Good for

  • Giving a coding agent live web search and page extraction inside an IDE
  • Scraping protected listing or job pages that block plain HTTP requests
  • Running UI or end-to-end test flows with route mocks and trace export through an agent
  • Compiling multi-source research summaries with optional local or hosted LLM synthesis

Pricing

Open source

MIT licensed, self-hosted.

ShadowCrawl FAQ

How is the server run?

The README says it is distributed as prebuilt binaries on GitHub Releases or built from source with Cargo, and that cortex-scout-mcp is the MCP stdio server while cortex-scout is an optional HTTP server.

Does it need a proxy?

No. The README says the default is direct, no-proxy traffic, and proxy support is enabled only when IP_LIST_PATH and PROXY_SOURCE_PATH are set.

Which MCP clients are covered in the setup guide?

The README gives configuration examples for VS Code, Cursor and Claude Desktop, with a multi-IDE guide in docs/IDE_SETUP.md.

Is ShadowCrawl free?

Yes. ShadowCrawl is open source, so you can use and self-host it for free.

What are the best alternatives to ShadowCrawl?

The closest alternatives to ShadowCrawl are acrawl, Scrapling MCP Server, Webshift, webclaw and Cat Burglar.

Where this comes from

  • Description: written from the project's README and its MCP Registry entry.

How AIPoweredBot checks servers