Skip to content

AI for web search and scraping

132 AI tools, agents and MCP servers for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–30 of 132

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. Command-line channels that let AI agents read X, Reddit, YouTube and Chinese platforms

    • Open source
    • Open-source agent
    • 112K GitHub stars
    • AI models and LLM tooling
  3. Browser automation CLI that drives your logged-in Chrome, with reusable site commands

    • Open source
    • Open-source agent
    • 31.1K GitHub stars
    • Browser automation
  4. Open-source research agent that writes cited reports from the web and your documents

    • Open source
    • Open-source agent
    • 30.7K GitHub stars
    • Research
  5. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  6. Browser engine and CLI built for agents that browse, extract and crawl at scale

    • Open source
    • Open-source agent
    • 1.2K GitHub stars
    • Browser automation
  7. Self-hosted browser agent that scrapes protected sites from a plain-English prompt

    • Open source
    • Open-source agent
    • 180 GitHub stars
    • Web search and scraping
  8. Search Console clicks, positions, URL inspection and sitemaps for your AI agent

    Runs locally
    • 8 tools
    • Open source
    • Web search and scraping
  9. ResearchZosho

    MCP server

    Local deep research assistant that checks each citation and keeps what it finds

    Runs locally
    • Open source
    • Knowledge and memory
  10. Fetch web pages as markdown with SSRF protection, reader mode, feeds and robots checks

    Runs locally
    • 15 tools
    • Open source
    • Web search and scraping
  11. Self-hosted Google Search Console MCP for analytics, URL inspection and sitemaps

    Not checked yet
    • Remote
    • Open source
    • Marketing and SEO
  12. Code Crawler

    MCP server

    Trace what really runs in Java, Python, TypeScript and JavaScript, with edge confidence

    Runs locally
    • 13 tools
    • Open source
    • Web search and scraping
  13. PyEuropePMC

    MCP server

    Europe PMC literature search, full text and export for AI agents (Python, MIT)

    Runs locally
    • Open source
    • Health and science
  14. Read-only Google Play data for agents: apps, charts, reviews, permissions and data safety

    Runs locally
    • 8 tools
    • Open source
    • E-commerce
  15. doc-scraper

    MCP server

    Crawl documentation sites into local Markdown that agents search offline

    Runs locally
    • 4 tools
    • Open source
    • Web search and scraping
  16. MCP Browser

    MCP server

    Browse login-protected and JavaScript-heavy pages through your own Chrome, Edge or Brave

    Runs locally
    • 9 tools
    • Open source
    • Browser automation
  17. FATSEARCH

    MCP server

    Turns web pages and PDFs into cached Markdown outlines read one section at a time

    Runs locally
    • Open source
    • Web search and scraping
  18. Remote MCP server that searches the MCP Harbor directory of MCP servers and agent tools

    Not checked yet
    • 2 tools
    • Remote
    • Open source
    • Developer tools
  19. OpenOSINT

    MCP server

    Passive-by-default OSINT lookups for AI agents: DNS, WHOIS, IP, breaches, Shodan and more

    Runs locally
    • 16 tools
    • Open source
    • Security
  20. Query your website change-tracking history and monitors in plain language

    Not checked yet
    • Remote
    • Open source
    • Productivity
  21. Local, read-only Google Search Console tools for traffic, indexing and sitemap checks

    Runs locally
    • 6 tools
    • Open source
    • Web search and scraping
  22. LastSearch

    MCP server

    Web research for agents that returns cited claims with confidence scores

    Runs locally
    • Open source
    • Web search and scraping
  23. Turn any web page into clean Markdown with one tool, run locally with no API key

    Runs locally
    • 1 tool
    • Open source
    • Files and documents
  24. Docker-run MCP server for finding public emails and subdomains of a domain

    Runs locally
    • 4 tools
    • Open source
    • Security
  25. WhatWeb MCP

    MCP server

    Docker-run MCP server that identifies the CMS, server and frameworks behind a website

    Runs locally
    • 3 tools
    • Open source
    • Security
  26. Free Search

    MCP server

    Keyless web search, page fetching and document reading that runs locally

    Runs locally
    • 11 tools
    • Open source
    • Web search and scraping
  27. Proxlane

    MCP server

    Web-fetch tool that returns block pages as failures, via your own Proxlane gateway

    Runs locally
    • 1 tool
    • Open source
    • Web search and scraping
  28. searxNcrawl

    MCP server

    Open-source MCP server for web search via SearXNG or You.com and Markdown crawling

    Runs locally
    • 3 tools
    • Open source
    • Web search and scraping
  29. Crawlberg

    MCP server

    Open-source crawler that turns websites into Markdown and JSON, with a local MCP server

    Runs locally
    • Open source
    • Web search and scraping
  30. shortfree

    MCP server

    Checks live whether short .com, .ai, .io and other domains are free; suggests free ones

    Runs locally
    • Open source
    • Developer tools