Skip to content
MCP server

Siftdog

Self-hosted, Tavily-compatible web search, extract, crawl and map tools for agents

Runs locallyStatus and history
Pricing
Open source
Platforms
self-hosted, CLI, Linux
Runs
Locally
Version
0.3.0
Listing updated
1 day ago

About Siftdog

Siftdog is an open-source MCP server for web search and scraping. It runs locally on your machine and is started by your AI client.

Siftdog reproduces the four Tavily endpoints (search, extract, crawl, map) on infrastructure you run, so code written for the official tavily-python SDK or LangChain works by changing only the base URL. Search goes through a SearXNG metasearch instance with BM25 relevance blended into the engines' order, so no search API key or per-search bill is needed.

Extraction produces clean markdown or text from up to 20 pages per request using trafilatura, and crawling follows links with depth, breadth, limit and regex filters. Optional answers can be written by Claude from retrieved results. The MCP server has four read-only tools over stdio or at /mcp on a running server. Private and internal addresses are blocked, including through redirects and DNS rebinding.

It fetches static HTML only, with no JavaScript rendering, and the site compares itself against Tavily and Firecrawl accordingly. A hosted version is described as planned. Python 3.11+, Docker images for amd64 and arm64, MIT licensed.

Install

Run in a terminal

claude mcp add siftdog -- uvx siftdog

Add to claude_desktop_config.json

{
  "mcpServers": {
    "siftdog": {
      "command": "uvx",
      "args": [
        "siftdog"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "siftdog": {
      "command": "uvx",
      "args": [
        "siftdog"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "siftdog": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "siftdog"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "siftdog": {
      "command": "uvx",
      "args": [
        "siftdog"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "siftdog": {
      "command": "uvx",
      "args": [
        "siftdog"
      ]
    }
  }
}

Configuration

  • SEARXNG_URLURL of a SearXNG instance with JSON output enabled. Needed only for siftdog_search; extract, crawl and map work without it.

Tools this server provides (4)

Taken from the project documentation; the running server may differ.

siftdog_search
Search the web through SearXNG.
siftdog_extract
Extract clean content from pages.
siftdog_crawl
Crawl a site by following links.
siftdog_map
List a site's URLs.

What it does

Tavily-compatible API
Same endpoints and response shapes, so existing SDK code keeps working.
SearXNG search
Ranked results with no search API key.
Page extraction
Markdown or text from up to 20 pages per request.
Crawl and map
Depth, breadth and regex filters, or just list URLs.
SSRF protection
Blocks private addresses on connect and every redirect.
Docker and pip
Run the full stack or only extract, crawl and map.

Good for

  • Replace a hosted search API with your own server
  • Give a coding agent web search without a vendor key
  • Crawl a documentation site into markdown
  • List all URLs of a site

Pricing

Open source

Prices change. Check siftdog.com before you buy.

Siftdog FAQ

Is it free?

The software is MIT licensed and self-hosted; you run SearXNG yourself.

Can it render JavaScript?

No, it fetches static HTML only.

Which tools does MCP expose?

siftdog_search, siftdog_extract, siftdog_crawl and siftdog_map.

Is Siftdog free?

Yes. Siftdog is open source, so you can use and self-host it for free.

How do I add Siftdog to Claude Code?

Run this command in your terminal: claude mcp add siftdog -- uvx siftdog

What are the best alternatives to Siftdog?

The closest alternatives to Siftdog are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.

Alternatives to Siftdog

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  4. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  5. Search AI tools, agents and MCP servers and get install configs from inside your AI assistant.

    • 4 tools
    • Remote
    • Free
    • Developer tools
  6. Hosted MCP server that returns any Shopify store's products, prices and stock

    Not checked yet
    • 4 tools
    • Remote
    • Paid
    • E-commerce