Skip to content
MCP server

Webshift

Rust MCP server that cleans and reranks web pages into compact text for AI agents

Pricing
Open source
Platforms
CLI, API, Linux
Version
0.2.9
Listing updated
today

Similar MCP servers

About Webshift

Webshift is an open-source MCP server for web search and scraping. It is available on CLI, API, Linux, macOS, Windows and self-hosted.

The README describes WebShift as a Rust library and MCP server that turns noisy HTML into clean, size-limited text for language models. Scripts, navigation, ads and cookie banners are stripped before the text reaches the model, so a context window is not filled with page chrome.

The MCP server, shipped as the mcp-webshift binary, exposes webshift_query for a full search, fetch, clean and BM25 rerank run, webshift_fetch for a single page, and webshift_onboarding for a usage guide. Search can go through SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing or a generic HTTP backend. LLM query expansion, summarization and reranking are optional and disabled by default.

The README targets developers who want a single-binary web search tool for agents with hard caps on download size, per-page length and total output. It notes that WebShift does not execute JavaScript, so it is not a replacement for a headless browser.

Tools this server provides (3)

Taken from the project documentation; the running server may differ.

webshift_query
Full search pipeline: search, fetch, clean, rerank and optional summarize.
webshift_fetch
Single page fetch and clean.
webshift_onboarding
Returns a JSON guide for the agent covering budgets, backends and tips.

What it does

HTML noise removal
Strips scripts, navigation, ads and similar page elements before returning plain text.
Budget-controlled output
Enforces per-page, total query and result-count limits so responses stay a predictable size.
Multiple search backends
Supports SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing and a generic HTTP backend.
BM25 reranking
Ranks fetched pages deterministically with BM25, with optional LLM-assisted second-tier reranking.
Binary URL filtering
Drops links to binary files such as PDF or ZIP before any network request is made.
Text-map rewriting
Extracts text nodes from HTML and rebuilds the markup with only the text replaced.

Good for

  • Giving an AI agent web search results as clean, size-capped text
  • Cleaning HTML in a Rust pipeline before it reaches an LLM
  • Translating or rewriting text inside HTML without breaking tags or links
  • Running a self-hosted MCP web search server with SearXNG and no API keys

Pricing

Open source

MIT licensed and free to use; search backend APIs may have their own pricing.

Webshift FAQ

Does WebShift render JavaScript-heavy pages?

No. The README says it parses static HTML and does not execute JavaScript, so a headless browser is needed for SPAs.

Which MCP tools does the server expose?

webshift_query, webshift_fetch and webshift_onboarding.

Is LLM use required?

No. The LLM features are opt-in and disabled by default; the README says no data leaves the machine unless they are enabled.

How is it installed?

With cargo install webshift-mcp, which produces the mcp-webshift binary.

Is Webshift free?

Yes. Webshift is open source, so you can use and self-host it for free.

What are the best alternatives to Webshift?

The closest alternatives to Webshift are Webshift, searchio, free-web-mcp, webclaw and Ransack.

Where this comes from

  • Description: written from the project's README and its MCP Registry entry.

How AIPoweredBot checks servers