# Webshift

> Webshift is an open-source MCP server for web search and scraping. Rust MCP server that cleans and reranks web pages into compact text for AI agents. It is available on CLI, API, Linux, macOS, Windows and self-hosted.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, API, Linux, macOS, Windows, self-hosted
- **Repository:** https://github.com/x-monk/webshift
- **MCP Registry name:** io.github.x-monk/webshift
- **Categories:** [web search and scraping](https://ai-powered.com/categories/web-search-and-scraping)
- **Listing updated:** 2026-11-18

## About

The README describes WebShift as a Rust library and MCP server that turns noisy HTML into clean, size-limited text for language models. Scripts, navigation, ads and cookie banners are stripped before the text reaches the model, so a context window is not filled with page chrome.

The MCP server, shipped as the mcp-webshift binary, exposes webshift_query for a full search, fetch, clean and BM25 rerank run, webshift_fetch for a single page, and webshift_onboarding for a usage guide. Search can go through SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing or a generic HTTP backend. LLM query expansion, summarization and reranking are optional and disabled by default.

The README targets developers who want a single-binary web search tool for agents with hard caps on download size, per-page length and total output. It notes that WebShift does not execute JavaScript, so it is not a replacement for a headless browser.

## Features

- **HTML noise removal:** Strips scripts, navigation, ads and similar page elements before returning plain text.
- **Budget-controlled output:** Enforces per-page, total query and result-count limits so responses stay a predictable size.
- **Multiple search backends:** Supports SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing and a generic HTTP backend.
- **BM25 reranking:** Ranks fetched pages deterministically with BM25, with optional LLM-assisted second-tier reranking.
- **Binary URL filtering:** Drops links to binary files such as PDF or ZIP before any network request is made.
- **Text-map rewriting:** Extracts text nodes from HTML and rebuilds the markup with only the text replaced.

## Good for

- Giving an AI agent web search results as clean, size-capped text
- Cleaning HTML in a Rust pipeline before it reaches an LLM
- Translating or rewriting text inside HTML without breaking tags or links
- Running a self-hosted MCP web search server with SearXNG and no API keys

## Tools

- `webshift_query`: Full search pipeline: search, fetch, clean, rerank and optional summarize.
- `webshift_fetch`: Single page fetch and clean.
- `webshift_onboarding`: Returns a JSON guide for the agent covering budgets, backends and tips.

## FAQ

### Does WebShift render JavaScript-heavy pages?

No. The README says it parses static HTML and does not execute JavaScript, so a headless browser is needed for SPAs.

### Which MCP tools does the server expose?

webshift_query, webshift_fetch and webshift_onboarding.

### Is LLM use required?

No. The LLM features are opt-in and disabled by default; the README says no data leaves the machine unless they are enabled.

### How is it installed?

With cargo install webshift-mcp, which produces the mcp-webshift binary.

### Is Webshift free?

Yes. Webshift is open source, so you can use and self-host it for free.

### What are the best alternatives to Webshift?

The closest alternatives to Webshift are Webshift, searchio, free-web-mcp, webclaw and Ransack.

## Where this comes from

- Description: written from the project's README and its MCP Registry entry.

## Alternatives

- [Webshift](https://ai-powered.com/mcp/webshift-3): Webshift is an open-source MCP server for web search and scraping.
- [searchio](https://ai-powered.com/mcp/searchio): searchio is an open-source MCP server for web search and scraping.
- [free-web-mcp](https://ai-powered.com/mcp/free-web-mcp): free-web-mcp is a free MCP server for web search and scraping.
- [webclaw](https://ai-powered.com/mcp/webclaw): webclaw is a freemium MCP server for web search and scraping.
- [Ransack](https://ai-powered.com/mcp/ransack): Ransack is an MCP server for web search and scraping.
- [Open Agent Search](https://ai-powered.com/mcp/open-agent-search): Open Agent Search is an open-source MCP server for web search and scraping.
- [Crawlberg](https://ai-powered.com/mcp/crawlberg): Crawlberg is an open-source MCP server for web search and scraping.
- [Free Search](https://ai-powered.com/mcp/free-search): Free Search is an open-source MCP server for web search and scraping.

---
Source: https://ai-powered.com/mcp/webshift-2
