Siftdog
Self-hosted, Tavily-compatible web search, extract, crawl and map tools for agents
- Pricing
- Open source
- Platforms
- self-hosted, CLI, Linux
- Runs
- Locally
- Version
- 0.3.0
- Listing updated
- 1 day ago
About Siftdog
Siftdog is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.
Siftdog reproduces the four Tavily endpoints (search, extract, crawl, map) on infrastructure you run, so code written for the official tavily-python SDK or LangChain works by changing only the base URL. Search goes through a SearXNG metasearch instance with BM25 relevance blended into the engines' order, so no search API key or per-search bill is needed.
Extraction produces clean markdown or text from up to 20 pages per request using trafilatura, and crawling follows links with depth, breadth, limit and regex filters. Optional answers can be written by Claude from retrieved results. The MCP server has four read-only tools over stdio or at /mcp on a running server. Private and internal addresses are blocked, including through redirects and DNS rebinding.
It fetches static HTML only, with no JavaScript rendering, and the site compares itself against Tavily and Firecrawl accordingly. A hosted version is described as planned. Python 3.11+, Docker images for amd64 and arm64, MIT licensed.
Install
Run in a terminal
claude mcp add siftdog -- uvx siftdogAdd to claude_desktop_config.json
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": [
"siftdog"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": [
"siftdog"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"siftdog": {
"type": "stdio",
"command": "uvx",
"args": [
"siftdog"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": [
"siftdog"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": [
"siftdog"
]
}
}
}Configuration
SEARXNG_URLURL of a SearXNG instance with JSON output enabled. Needed only for siftdog_search; extract, crawl and map work without it.
Tools this server provides (4)
Taken from the project documentation; the running server may differ.
- siftdog_search
- Search the web through SearXNG.
- siftdog_extract
- Extract clean content from pages.
- siftdog_crawl
- Crawl a site by following links.
- siftdog_map
- List a site's URLs.
What it does
- Tavily-compatible API
- Same endpoints and response shapes, so existing SDK code keeps working.
- SearXNG search
- Ranked results with no search API key.
- Page extraction
- Markdown or text from up to 20 pages per request.
- Crawl and map
- Depth, breadth and regex filters, or just list URLs.
- SSRF protection
- Blocks private addresses on connect and every redirect.
- Docker and pip
- Run the full stack or only extract, crawl and map.
Good for
- Replace a hosted search API with your own server
- Give a coding agent web search without a vendor key
- Crawl a documentation site into markdown
- List all URLs of a site
Pricing
Open source
Prices change. Check siftdog.com before you buy.
Siftdog FAQ
Is it free?
The software is MIT licensed and self-hosted; you run SearXNG yourself.
Can it render JavaScript?
No, it fetches static HTML only.
Which tools does MCP expose?
siftdog_search, siftdog_extract, siftdog_crawl and siftdog_map.
Is Siftdog free?
Yes. Siftdog is open source, so you can use and self-host it for free.
How do I add Siftdog to Claude Code?
Run this command in your terminal: claude mcp add siftdog -- uvx siftdog
What are the best alternatives to Siftdog?
The closest alternatives to Siftdog are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.
Alternatives to Siftdog

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping

AI-Powered directory
MCP serverSearch AI tools, agents and MCP servers and get install configs from inside your AI assistant.
- 4 tools
- Remote
- Free
- Developer tools

Shopify MCP Server (any store)
MCP serverHosted MCP server that returns any Shopify store's products, prices and stock
Not checked yet- 4 tools
- Remote
- Paid
- E-commerce