Skip to content
MCP server

Webshift

Rust MCP server that fetches, cleans and reranks web pages into compact text for AI agents

Pricing
Open source
Platforms
CLI, self-hosted, API
Version
0.2.8
Listing updated
today

Similar MCP servers

About Webshift

Webshift is an open-source MCP server for web search and scraping. It is available on CLI, self-hosted and API.

The README says WebShift turns noisy HTML into compact, size-limited text so that an LLM does not spend its context window on scripts, navigation and cookie banners. It ships as a single Rust binary, mcp-webshift, that speaks MCP over stdio and exposes search, single-page fetch and an onboarding guide to the agent.

A query runs through a pipeline: optional LLM query expansion, a search call to one of several backends, deduplication and filtering of binary URLs, streaming fetches with a per-page size cap, HTML cleaning, Unicode and BiDi sterilization, BM25 reranking and budget-aware truncation. Output is structured JSON, and an optional LLM step can produce a Markdown summary with inline citations.

The project is aimed at developers building agents who want predictable output sizes and a search tool that needs no Python or Docker setup. It is a static HTML pipeline, so it does not execute JavaScript or handle logins and forms, which the README lists as out of scope.

Tools this server provides (3)

Taken from the project documentation; the running server may differ.

webshift_query
Full search pipeline: search, fetch, clean, rerank and optionally summarize.
webshift_fetch
Single page fetch and clean.
webshift_onboarding
Returns a JSON guide for the agent covering budgets, backends and tips.

What it does

Noise removal from HTML
Strips scripts, navigation, ads and similar page elements before the text reaches the model.
Hard output size caps
Per-page, total-budget and result-count limits keep responses within a set character budget.
Streaming download limit
Caps how much of each page is downloaded, so large responses are not buffered in full.
BM25 reranking
Orders fetched sources by relevance with a deterministic BM25 pass, with optional LLM reranking on top.
Multiple search backends
Works with SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing or a generic HTTP search endpoint.
Optional LLM features
Query expansion and Markdown summaries with citations are opt-in and work with OpenAI-compatible APIs.

Good for

  • Give an AI agent web search results as clean, budget-controlled text
  • Clean scraped HTML in a Rust pipeline before sending it to an LLM
  • Translate or rewrite text inside HTML while keeping the markup intact
  • Run a single-binary MCP web search server without Python or Docker

Pricing

Open source

MIT licensed; free to self-host. SearXNG is free and self-hosted; some cloud search backends have free tiers with limits.

Webshift FAQ

Does WebShift render JavaScript-heavy pages?

No. The README says it parses static HTML and does not execute JavaScript.

Which search backends are supported?

SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing and a generic HTTP backend configurable in TOML.

Does it send data to an LLM by default?

No. The README says LLM features are opt-in and disabled by default.

What tools does the MCP server expose?

webshift_query, webshift_fetch and webshift_onboarding.

Is Webshift free?

Yes. Webshift is open source, so you can use and self-host it for free.

What are the best alternatives to Webshift?

The closest alternatives to Webshift are Webshift, free-web-mcp, searchio, Open Agent Search and webclaw.

Where this comes from

  • Description: written from the project's README and its MCP Registry entry.

How AIPoweredBot checks servers