Webshift
Rust MCP server that fetches, cleans and reranks web pages into compact text for AI agents
- Pricing
- Open source
- Platforms
- CLI, self-hosted, API
- Version
- 0.2.8
- Listing updated
- today
Similar MCP servers
About Webshift
Webshift is an open-source MCP server for web search and scraping. It is available on CLI, self-hosted and API.
The README says WebShift turns noisy HTML into compact, size-limited text so that an LLM does not spend its context window on scripts, navigation and cookie banners. It ships as a single Rust binary, mcp-webshift, that speaks MCP over stdio and exposes search, single-page fetch and an onboarding guide to the agent.
A query runs through a pipeline: optional LLM query expansion, a search call to one of several backends, deduplication and filtering of binary URLs, streaming fetches with a per-page size cap, HTML cleaning, Unicode and BiDi sterilization, BM25 reranking and budget-aware truncation. Output is structured JSON, and an optional LLM step can produce a Markdown summary with inline citations.
The project is aimed at developers building agents who want predictable output sizes and a search tool that needs no Python or Docker setup. It is a static HTML pipeline, so it does not execute JavaScript or handle logins and forms, which the README lists as out of scope.
Tools this server provides (3)
Taken from the project documentation; the running server may differ.
- webshift_query
- Full search pipeline: search, fetch, clean, rerank and optionally summarize.
- webshift_fetch
- Single page fetch and clean.
- webshift_onboarding
- Returns a JSON guide for the agent covering budgets, backends and tips.
What it does
- Noise removal from HTML
- Strips scripts, navigation, ads and similar page elements before the text reaches the model.
- Hard output size caps
- Per-page, total-budget and result-count limits keep responses within a set character budget.
- Streaming download limit
- Caps how much of each page is downloaded, so large responses are not buffered in full.
- BM25 reranking
- Orders fetched sources by relevance with a deterministic BM25 pass, with optional LLM reranking on top.
- Multiple search backends
- Works with SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing or a generic HTTP search endpoint.
- Optional LLM features
- Query expansion and Markdown summaries with citations are opt-in and work with OpenAI-compatible APIs.
Good for
- Give an AI agent web search results as clean, budget-controlled text
- Clean scraped HTML in a Rust pipeline before sending it to an LLM
- Translate or rewrite text inside HTML while keeping the markup intact
- Run a single-binary MCP web search server without Python or Docker
Pricing
Open source
MIT licensed; free to self-host. SearXNG is free and self-hosted; some cloud search backends have free tiers with limits.
Webshift FAQ
Does WebShift render JavaScript-heavy pages?
No. The README says it parses static HTML and does not execute JavaScript.
Which search backends are supported?
SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing and a generic HTTP backend configurable in TOML.
Does it send data to an LLM by default?
No. The README says LLM features are opt-in and disabled by default.
What tools does the MCP server expose?
webshift_query, webshift_fetch and webshift_onboarding.
Is Webshift free?
Yes. Webshift is open source, so you can use and self-host it for free.
What are the best alternatives to Webshift?
The closest alternatives to Webshift are Webshift, free-web-mcp, searchio, Open Agent Search and webclaw.
Where this comes from
- Description: written from the project's README and its MCP Registry entry.
