# Webshift

> Webshift is an open-source MCP server for web search and scraping. Rust MCP server that fetches, cleans and reranks web pages into compact text for AI agents. It is available on CLI, self-hosted and API.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, self-hosted, API
- **Repository:** https://github.com/annibale-x/webshift
- **MCP Registry name:** io.github.annibale-x/webshift
- **Categories:** [web search and scraping](https://ai-powered.com/categories/web-search-and-scraping)
- **Listing updated:** 2026-11-18

## About

The README says WebShift turns noisy HTML into compact, size-limited text so that an LLM does not spend its context window on scripts, navigation and cookie banners. It ships as a single Rust binary, mcp-webshift, that speaks MCP over stdio and exposes search, single-page fetch and an onboarding guide to the agent.

A query runs through a pipeline: optional LLM query expansion, a search call to one of several backends, deduplication and filtering of binary URLs, streaming fetches with a per-page size cap, HTML cleaning, Unicode and BiDi sterilization, BM25 reranking and budget-aware truncation. Output is structured JSON, and an optional LLM step can produce a Markdown summary with inline citations.

The project is aimed at developers building agents who want predictable output sizes and a search tool that needs no Python or Docker setup. It is a static HTML pipeline, so it does not execute JavaScript or handle logins and forms, which the README lists as out of scope.

## Features

- **Noise removal from HTML:** Strips scripts, navigation, ads and similar page elements before the text reaches the model.
- **Hard output size caps:** Per-page, total-budget and result-count limits keep responses within a set character budget.
- **Streaming download limit:** Caps how much of each page is downloaded, so large responses are not buffered in full.
- **BM25 reranking:** Orders fetched sources by relevance with a deterministic BM25 pass, with optional LLM reranking on top.
- **Multiple search backends:** Works with SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing or a generic HTTP search endpoint.
- **Optional LLM features:** Query expansion and Markdown summaries with citations are opt-in and work with OpenAI-compatible APIs.

## Good for

- Give an AI agent web search results as clean, budget-controlled text
- Clean scraped HTML in a Rust pipeline before sending it to an LLM
- Translate or rewrite text inside HTML while keeping the markup intact
- Run a single-binary MCP web search server without Python or Docker

## Tools

- `webshift_query`: Full search pipeline: search, fetch, clean, rerank and optionally summarize.
- `webshift_fetch`: Single page fetch and clean.
- `webshift_onboarding`: Returns a JSON guide for the agent covering budgets, backends and tips.

## FAQ

### Does WebShift render JavaScript-heavy pages?

No. The README says it parses static HTML and does not execute JavaScript.

### Which search backends are supported?

SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing and a generic HTTP backend configurable in TOML.

### Does it send data to an LLM by default?

No. The README says LLM features are opt-in and disabled by default.

### What tools does the MCP server expose?

webshift_query, webshift_fetch and webshift_onboarding.

### Is Webshift free?

Yes. Webshift is open source, so you can use and self-host it for free.

### What are the best alternatives to Webshift?

The closest alternatives to Webshift are Webshift, free-web-mcp, searchio, Open Agent Search and webclaw.

## Where this comes from

- Description: written from the project's README and its MCP Registry entry.

## Alternatives

- [Webshift](https://ai-powered.com/mcp/webshift-2): Webshift is an open-source MCP server for web search and scraping.
- [free-web-mcp](https://ai-powered.com/mcp/free-web-mcp): free-web-mcp is a free MCP server for web search and scraping.
- [searchio](https://ai-powered.com/mcp/searchio): searchio is an open-source MCP server for web search and scraping.
- [Open Agent Search](https://ai-powered.com/mcp/open-agent-search): Open Agent Search is an open-source MCP server for web search and scraping.
- [webclaw](https://ai-powered.com/mcp/webclaw): webclaw is a freemium MCP server for web search and scraping.
- [MarkUDown](https://ai-powered.com/mcp/markudown): MarkUDown is a freemium MCP server for web search and scraping.
- [URLpipe](https://ai-powered.com/mcp/urlpipe): URLpipe is a freemium MCP server for web search and scraping.
- [machine_internet](https://ai-powered.com/mcp/machine-internet): machine_internet is an open-source MCP server for web search and scraping.

---
Source: https://ai-powered.com/mcp/webshift-3
