# Webshift

> Webshift is an open-source MCP server for web search and scraping. Rust MCP server that fetches, cleans and reranks web search results for AI agents. It is available on CLI, API, Linux, macOS, Windows and self-hosted.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, API, Linux, macOS, Windows, self-hosted
- **Repository:** https://github.com/x-hannibal/webshift
- **MCP Registry name:** io.github.x-hannibal/webshift
- **Categories:** [web search and scraping](https://ai-powered.com/categories/web-search-and-scraping)
- **Listing updated:** 2026-11-18

## About

The README says WebShift is a Rust library and MCP server aimed at turning noisy web pages into compact text that fits within an LLM's context budget. It removes scripts, navigation, banners and similar clutter, strips problematic Unicode control characters, and then reranks the surviving pages with BM25 before applying size limits across all sources.

The MCP server ships as a single binary called mcp-webshift and exposes webshift_query, webshift_fetch and webshift_onboarding over stdio. Search can go through SearXNG (self-hosted, no key), Brave, Tavily, Exa, SerpAPI, Google, Bing or a generic HTTP backend. LLM query expansion, summarization and reranking are optional and disabled by default.

The project is aimed at developers building agents or Rust pipelines that need budget-controlled web text. The README is explicit that it parses static HTML only and does not run JavaScript, render pages or handle logins, so it is a poor fit for SPA-heavy sites or browser automation. It is still in beta.

## Features

- **Noise removal and cleanup:** Strips scripts, navigation, ads and cookie banners from HTML and returns plain text.
- **BM25 reranking:** Orders fetched pages with deterministic BM25 scoring, with optional LLM-assisted second-tier reranking.
- **Hard size budgets:** Applies per-page character caps, total query budgets and streaming download limits.
- **Multiple search backends:** Supports SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing and a configurable HTTP backend.
- **Binary filtering:** Skips PDFs, archives and executables before making any network request.
- **Rust library API:** Exposes clean, fetch and query functions, plus text-map round-tripping of HTML text nodes.

## Good for

- Giving an AI agent web search results as clean, size-limited text
- Cleaning raw HTML in a Rust data pipeline before passing it to an LLM
- Translating or rewriting page text while keeping markup and links intact
- Running a self-hosted web search tool for an MCP client without cloud API keys

## Tools

- `webshift_query`: Full search pipeline: search, fetch, clean, rerank and optionally summarize.
- `webshift_fetch`: Single page fetch and clean.
- `webshift_onboarding`: Returns a JSON guide for the agent covering budgets, backends and tips.

## FAQ

### Does WebShift render JavaScript-heavy pages?

No. The README says it parses static HTML and does not execute JavaScript.

### Which search backend is free and needs no API key?

The README names SearXNG as free, self-hosted and keyless, run for example via Docker on port 8080.

### What MCP tools does the server expose?

webshift_query, webshift_fetch and webshift_onboarding.

### What license is it under?

The README states the MIT License.

### Is Webshift free?

Yes. Webshift is open source, so you can use and self-host it for free.

### What are the best alternatives to Webshift?

The closest alternatives to Webshift are searchio, free-web-mcp, webclaw, Open Agent Search and Ransack.

## Where this comes from

- Description: written from the project's README and its MCP Registry entry.

## Alternatives

- [searchio](https://ai-powered.com/mcp/searchio): searchio is an open-source MCP server for web search and scraping.
- [free-web-mcp](https://ai-powered.com/mcp/free-web-mcp): free-web-mcp is a free MCP server for web search and scraping.
- [webclaw](https://ai-powered.com/mcp/webclaw): webclaw is a freemium MCP server for web search and scraping.
- [Open Agent Search](https://ai-powered.com/mcp/open-agent-search): Open Agent Search is an open-source MCP server for web search and scraping.
- [Ransack](https://ai-powered.com/mcp/ransack): Ransack is an MCP server for web search and scraping.
- [SearchPipe](https://ai-powered.com/mcp/searchpipe): SearchPipe is a freemium MCP server for web search and scraping.
- [MarkUDown](https://ai-powered.com/mcp/markudown): MarkUDown is a freemium MCP server for web search and scraping.
- [Visdenda Free MCP](https://ai-powered.com/mcp/visdenda-free-mcp): Visdenda Free MCP is a free MCP server for web search and scraping.

---
Source: https://ai-powered.com/mcp/webshift
