Skip to content

AI for web search and scraping

60 AI tools, agents and MCP servers for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–30 of 60

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. Web search, scraping and page interaction API for AI agents and developers

    • Freemium
    • Agent framework
    • 202K GitHub stars
    • Web search and scraping
  3. Web crawler and hosted API that turns URLs into LLM-ready Markdown or JSON

    • Freemium
    • Open-source agent
    • 87.5K GitHub stars
    • Web search and scraping
  4. Open-source research agent that writes cited reports from the web and your documents

    • Open source
    • Open-source agent
    • 30.7K GitHub stars
    • Research
  5. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  6. Browser engine and CLI built for agents that browse, extract and crawl at scale

    • Open source
    • Open-source agent
    • 1.2K GitHub stars
    • Browser automation
  7. Self-hosted browser agent that scrapes protected sites from a plain-English prompt

    • Open source
    • Open-source agent
    • 180 GitHub stars
    • Web search and scraping
  8. Hosted and open-source web scraper that returns clean Markdown with change tracking

    • Remote
    • Free trial
    • Web search and scraping
  9. ResearchZosho

    MCP server

    Local deep research assistant that checks each citation and keeps what it finds

    Runs locally
    • Open source
    • Knowledge and memory
  10. SerpApi

    MCP server

    Official SerpApi server: Google, Bing and other search results as JSON or Markdown

    • 4 tools
    • Remote
    • Freemium
    • Web search and scraping
  11. CRW Web Scraper

    MCP server

    Open-source web scraper for agents with scrape, crawl, map and search tools

    Runs locally
    • 3 tools
    • Freemium
    • Web search and scraping
  12. Octocrawl

    MCP server

    Web scraping, mapping and crawling MCP server that returns clean Markdown

    Not checked yet
    • 2 tools
    • Remote
    • Freemium
    • Web search and scraping
  13. Self-hosted Google Search Console MCP for analytics, URL inspection and sitemaps

    Not checked yet
    • Remote
    • Open source
    • Marketing and SEO
  14. OmniSeek

    MCP server

    Search podcasts, videos, Chinese forums and paper citations from your coding agent

    Not checked yet
    • Remote
    • Free
    • Web search and scraping
  15. Code Crawler

    MCP server

    Trace what really runs in Java, Python, TypeScript and JavaScript, with edge confidence

    Runs locally
    • 13 tools
    • Open source
    • Web search and scraping
  16. Seven website audit tools for agents, billed per call through your own Apify account

    Not checked yet
    • 7 tools
    • Remote
    • Paid
    • Marketing and SEO
  17. Crawlora MCP

    MCP server

    Hosted MCP exposing 4,684 structured web-data tools across 534 platforms, returned as JSON

    Not checked yet
    • Remote
    • Freemium
    • Web search and scraping
  18. Search immunology and virology literature, trials and grants across six databases at once

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Health and science
  19. PyEuropePMC

    MCP server

    Europe PMC literature search, full text and export for AI agents (Python, MIT)

    Runs locally
    • Open source
    • Health and science
  20. Read-only Google Play data for agents: apps, charts, reviews, permissions and data safety

    Runs locally
    • 8 tools
    • Open source
    • E-commerce
  21. Read-only Search Console data for assistants: queries, indexing, sitemaps

    Runs locally
    • Freemium
    • Web search and scraping
  22. Pay-per-call B2B leads, clean Markdown and e-commerce prices for AI agents

    Not checked yet
    • 4 tools
    • Remote
    • Paid
    • Web search and scraping
  23. doc-scraper

    MCP server

    Crawl documentation sites into local Markdown that agents search offline

    Runs locally
    • 4 tools
    • Open source
    • Web search and scraping
  24. Pay-per-call catalog of 500+ agent tools, paid in USDC over x402 or free by proof-of-work

    Not checked yet
    • Remote
    • Freemium
    • Web search and scraping
  25. Scrape.ly

    MCP server

    Find, run and read results from Scrape.ly Actors inside any MCP client

    Not checked yet
    • 14 tools
    • Remote
    • Freemium
    • Web search and scraping
  26. Remote MCP server that searches the MCP Harbor directory of MCP servers and agent tools

    Not checked yet
    • 2 tools
    • Remote
    • Open source
    • Developer tools
  27. OpenOSINT

    MCP server

    Passive-by-default OSINT lookups for AI agents: DNS, WHOIS, IP, breaches, Shodan and more

    Runs locally
    • 16 tools
    • Open source
    • Security
  28. Query your website change-tracking history and monitors in plain language

    Not checked yet
    • Remote
    • Open source
    • Productivity
  29. webclaw

    MCP server

    Scrape, crawl, search and extract web pages as clean Markdown or JSON for agents

    Not checked yet
    • Remote
    • Freemium
    • Web search and scraping
  30. Turn any web page into clean Markdown with one tool, run locally with no API key

    Runs locally
    • 1 tool
    • Open source
    • Files and documents