Skip to content

AI for web search and scraping

28 AI tools, agents and MCP servers for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–28 of 28

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. ClawGrab

    MCP server

    Fetch transcripts of public YouTube, TikTok, Vimeo and podcast links for AI use

    Runs locally
    • 6 tools
    • Freemium
    • Web search and scraping
  3. proxy-scraper

    MCP server

    Hourly-checked free HTTP and SOCKS proxy lists with a CLI and MCP tools to fetch pages

    Runs locally
    • Free
    • Web search and scraping
  4. OmniSeek

    MCP server

    Search podcasts, videos, Chinese forums and paper citations from your coding agent

    Not checked yet
    • Remote
    • Free
    • Web search and scraping
  5. Read-only Google Play data for agents: apps, charts, reviews, permissions and data safety

    Runs locally
    • 8 tools
    • Open source
    • E-commerce
  6. doc-scraper

    MCP server

    Crawl documentation sites into local Markdown that agents search offline

    Runs locally
    • 4 tools
    • Open source
    • Web search and scraping
  7. FATSEARCH

    MCP server

    Turns web pages and PDFs into cached Markdown outlines read one section at a time

    Runs locally
    • Open source
    • Web search and scraping
  8. OpenOSINT

    MCP server

    Passive-by-default OSINT lookups for AI agents: DNS, WHOIS, IP, breaches, Shodan and more

    Runs locally
    • 16 tools
    • Open source
    • Security
  9. Local, read-only Google Search Console tools for traffic, indexing and sitemap checks

    Runs locally
    • 6 tools
    • Open source
    • Web search and scraping
  10. Serper.live

    MCP server

    Live Google results and domain rank checks by country and device, pay per search

    Runs locally
    • 3 tools
    • Freemium
    • Web search and scraping
  11. webclaw

    MCP server

    Scrape, crawl, search and extract web pages as clean Markdown or JSON for agents

    Not checked yet
    • Remote
    • Freemium
    • Web search and scraping
  12. Drive the SEO Content Machine desktop app: crawls, scraping, Maps and workspace pages

    Runs locally
    • 23 tools
    • Free trial
    • Marketing and SEO
  13. Docker-run MCP server for finding public emails and subdomains of a domain

    Runs locally
    • 4 tools
    • Open source
    • Security
  14. WhatWeb MCP

    MCP server

    Docker-run MCP server that identifies the CMS, server and frameworks behind a website

    Runs locally
    • 3 tools
    • Open source
    • Security
  15. Free Search

    MCP server

    Keyless web search, page fetching and document reading that runs locally

    Runs locally
    • 11 tools
    • Open source
    • Web search and scraping
  16. Crawl Readiness

    MCP server

    Check which AI crawlers can read a site and generate llms.txt, robots.txt and schema

    Runs locally
    • 9 tools
    • Freemium
    • Web search and scraping
  17. Comment Scraper

    MCP server

    Ask Claude about Reddit and Hacker News threads you exported with Comment Scraper

    Runs locally
    • 4 tools
    • Freemium
    • Web search and scraping
  18. Crawlberg

    MCP server

    Open-source crawler that turns websites into Markdown and JSON, with a local MCP server

    Runs locally
    • Open source
    • Web search and scraping
  19. Local research workspace that tracks sources, claims and gaps for your AI agent

    Runs locally
    • 5 tools
    • Free
    • Web search and scraping
  20. One AI interface to Search Console, Bing, GA4 and AdSense data, with multi-account support

    Runs locally
    • 5 tools
    • Open source
    • Web search and scraping
  21. FrankenSurf

    MCP server

    Reads web pages with the cheapest tool that works, then checks the page is real

    Runs locally
    • Free
    • Web search and scraping
  22. DomainScout

    MCP server

    Bulk domain availability checker for AI assistants, with prices at nine registrars

    Runs locally
    • 1 tool
    • Open source
    • Web search and scraping
  23. Self-hosted SEO crawler with AI-crawler checks that assistants can drive over MCP

    Runs locally
    • Open source
    • Web search and scraping
  24. KitchenSink4Web

    MCP server

    Budgeted web page reading for AI assistants, read-only until you allow clicking

    Runs locally
    • 1 tool
    • Open source
    • Browser automation
  25. Keyless web, news, image, video and book search for AI agents, via MCP or REST

    Runs locally
    • Open source
    • Web search and scraping
  26. Nimrod

    MCP server

    Google search with dates and filters, plus clean page and PDF reading for AI apps

    Not checked yet
    • 5 tools
    • Remote
    • Free trial
    • Web search and scraping
  27. Siftdog

    MCP server

    Self-hosted, Tavily-compatible web search, extract, crawl and map tools for agents

    Runs locally
    • 4 tools
    • Open source
    • Web search and scraping