Webshift
Rust MCP server that cleans and reranks web pages into compact text for AI agents
- Pricing
- Open source
- Platforms
- CLI, API, Linux
- Version
- 0.2.9
- Listing updated
- today
Similar MCP servers
About Webshift
Webshift is an open-source MCP server for web search and scraping. It is available on CLI, API, Linux, macOS, Windows and self-hosted.
The README describes WebShift as a Rust library and MCP server that turns noisy HTML into clean, size-limited text for language models. Scripts, navigation, ads and cookie banners are stripped before the text reaches the model, so a context window is not filled with page chrome.
The MCP server, shipped as the mcp-webshift binary, exposes webshift_query for a full search, fetch, clean and BM25 rerank run, webshift_fetch for a single page, and webshift_onboarding for a usage guide. Search can go through SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing or a generic HTTP backend. LLM query expansion, summarization and reranking are optional and disabled by default.
The README targets developers who want a single-binary web search tool for agents with hard caps on download size, per-page length and total output. It notes that WebShift does not execute JavaScript, so it is not a replacement for a headless browser.
Tools this server provides (3)
Taken from the project documentation; the running server may differ.
- webshift_query
- Full search pipeline: search, fetch, clean, rerank and optional summarize.
- webshift_fetch
- Single page fetch and clean.
- webshift_onboarding
- Returns a JSON guide for the agent covering budgets, backends and tips.
What it does
- HTML noise removal
- Strips scripts, navigation, ads and similar page elements before returning plain text.
- Budget-controlled output
- Enforces per-page, total query and result-count limits so responses stay a predictable size.
- Multiple search backends
- Supports SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing and a generic HTTP backend.
- BM25 reranking
- Ranks fetched pages deterministically with BM25, with optional LLM-assisted second-tier reranking.
- Binary URL filtering
- Drops links to binary files such as PDF or ZIP before any network request is made.
- Text-map rewriting
- Extracts text nodes from HTML and rebuilds the markup with only the text replaced.
Good for
- Giving an AI agent web search results as clean, size-capped text
- Cleaning HTML in a Rust pipeline before it reaches an LLM
- Translating or rewriting text inside HTML without breaking tags or links
- Running a self-hosted MCP web search server with SearXNG and no API keys
Pricing
Open source
MIT licensed and free to use; search backend APIs may have their own pricing.
Webshift FAQ
Does WebShift render JavaScript-heavy pages?
No. The README says it parses static HTML and does not execute JavaScript, so a headless browser is needed for SPAs.
Which MCP tools does the server expose?
webshift_query, webshift_fetch and webshift_onboarding.
Is LLM use required?
No. The LLM features are opt-in and disabled by default; the README says no data leaves the machine unless they are enabled.
How is it installed?
With cargo install webshift-mcp, which produces the mcp-webshift binary.
Is Webshift free?
Yes. Webshift is open source, so you can use and self-host it for free.
What are the best alternatives to Webshift?
The closest alternatives to Webshift are Webshift, searchio, free-web-mcp, webclaw and Ransack.
Where this comes from
- Description: written from the project's README and its MCP Registry entry.
