Skip to content

AI for web search and scraping

529 AI tools, agents, MCP servers and skills for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–30 of 529

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. Web search, scraping and page interaction API for AI agents and developers

    • Freemium
    • Agent framework
    • 202K GitHub stars
    • Web search and scraping
  3. Command-line channels that let AI agents read X, Reddit, YouTube and Chinese platforms

    • Open source
    • Open-source agent
    • 112K GitHub stars
    • AI models and LLM tooling
  4. Web crawler and hosted API that turns URLs into LLM-ready Markdown or JSON

    • Freemium
    • Open-source agent
    • 87.5K GitHub stars
    • Web search and scraping
  5. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  6. Browser automation CLI that drives your logged-in Chrome, with reusable site commands

    • Open source
    • Open-source agent
    • 31.1K GitHub stars
    • Browser automation
  7. Open-source research agent that writes cited reports from the web and your documents

    • Open source
    • Open-source agent
    • 30.7K GitHub stars
    • Research
  8. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  9. Browser engine and CLI built for agents that browse, extract and crawl at scale

    • Open source
    • Open-source agent
    • 1.2K GitHub stars
    • Browser automation
  10. Guides Qdrant monitoring setup including Prometheus scraping, health probes, Hybrid Cloud metrics, alerting, and log centralization

    • Open source
    • 41.3K GitHub stars
    • Knowledge and memory
  11. Build GitHub Copilot workflows with Xquik X API SDKs, REST endpoints, hosted Apify Actor runs, MCP tools, TweetClaw OpenClaw plugin installs

    • Open source
    • 41.3K GitHub stars
    • Web search and scraping
  12. Bing Webmaster Tools + IndexNow extension

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  13. Full-site crawling, scraping, and site mapping via Firecrawl MCP

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  14. SEO geo

    Skill

    Audit and improve content for AI Overviews and answer engines, including citability, entity clarity, crawler access, brand signals, and pass

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  15. Web API for agents: scrape, search, cloud browsers and task agents, billed in credits

    • Freemium
    • Open-source agent
    • 745 GitHub stars
    • Web search and scraping
  16. Self-hosted browser agent that scrapes protected sites from a plain-English prompt

    • Open source
    • Open-source agent
    • 180 GitHub stars
    • Web search and scraping
  17. Use when the user wants to fact-check an article or verify factual claims in a document

    • Open source
    • 770 GitHub stars
    • Web search and scraping
  18. Generates Puppeteer scripts for browser automation, scraping, and PDF generation

    • Open source
    • 392 GitHub stars
    • Browser automation
  19. Guides Qdrant monitoring setup including Prometheus scraping, Datadog integration, health probes, Hybrid Cloud metrics, alerting, and log ce

    • Open source
    • 269 GitHub stars
    • Knowledge and memory
  20. Multi-level research: built-in knowledge → web search → Tavily deep research API

    • Open source
    • 107 GitHub stars
    • Web search and scraping
  21. Use Exa for semantic/neural web search

    • Open source
    • 72 GitHub stars
    • Web search and scraping
  22. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  23. easyserp

    MCP server

    Structured Google and Bing search results for agents, with recoverable requests

    Runs locally
    • Free trial
    • Web search and scraping
  24. ChangePing

    MCP server

    Watch web pages from Claude or ChatGPT and read summaries of what changed

    Not checked yet
    • 7 tools
    • Remote
    • Freemium
    • Web search and scraping
  25. ClawGrab

    MCP server

    Fetch transcripts of public YouTube, TikTok, Vimeo and podcast links for AI use

    Runs locally
    • 6 tools
    • Freemium
    • Web search and scraping
  26. Pay-per-call tools for chain data, sanctions screening, web reading and PDFs

    • 15 tools
    • Remote
    • Freemium
    • Web search and scraping
  27. Hosted MCP server for capturing screenshots and page text through ApiFlash

    • 3 tools
    • Remote
    • Freemium
    • Browser automation
  28. Fetch pages and search the web through a Singapore 4G/5G mobile IP from your agent

    • 4 tools
    • Remote
    • Free trial
    • Web search and scraping
  29. maviapi

    MCP server

    Live JSON from sites with no API, plus agent-built APIs for any website

    Runs locally
    • 9 tools
    • Free trial
    • Web search and scraping
  30. Turn YouTube captions, PDFs and web pages into clean Markdown for RAG

    • Remote
    • Freemium
    • Knowledge and memory