Skip to content

AI for web search and scraping

392 AI tools, agents and MCP servers for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–30 of 392

  1. Web search, scraping and page interaction API for AI agents and developers

    • Freemium
    • Agent framework
    • 202K GitHub stars
    • Web search and scraping
  2. Web crawler and hosted API that turns URLs into LLM-ready Markdown or JSON

    • Freemium
    • Open-source agent
    • 87.5K GitHub stars
    • Web search and scraping
  3. Open-source research agent that writes cited reports from the web and your documents

    • Open source
    • Open-source agent
    • 30.7K GitHub stars
    • Research
  4. Web API for agents: scrape, search, cloud browsers and task agents, billed in credits

    • Freemium
    • Open-source agent
    • 745 GitHub stars
    • Web search and scraping
  5. Self-hosted browser agent that scrapes protected sites from a plain-English prompt

    • Open source
    • Open-source agent
    • 180 GitHub stars
    • Web search and scraping
  6. API Pick

    MCP server

    27 pay-per-call search tools: web, news, SEC, patents, clinical and more

    • 27 tools
    • Remote
    • Paid
    • Web search and scraping
  7. ChangePing

    MCP server

    Watch web pages from Claude or ChatGPT and read summaries of what changed

    Not checked yet
    • 7 tools
    • Remote
    • Freemium
    • Web search and scraping
  8. ClawGrab

    MCP server

    Fetch transcripts of public YouTube, TikTok, Vimeo and podcast links for AI use

    Runs locally
    • 6 tools
    • Freemium
    • Web search and scraping
  9. llms.ink

    MCP server

    Turn any website, GitHub repo, PDF or video link into llms.txt or llms-full.txt

    • 7 tools
    • Remote
    • Free
    • AI models and LLM tooling
  10. x-toolkit

    MCP server

    Read public X Articles, video transcripts and posts as plain text, no account needed

    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  11. CheckAnySite

    MCP server

    Ask an agent what a website is: AI builder traces, tech stack, traffic rank and trust

    • 6 tools
    • Remote
    • Free
    • Analytics and data
  12. Turn YouTube captions, PDFs and web pages into clean Markdown for RAG

    • Remote
    • Freemium
    • Knowledge and memory
  13. Read Bluesky, Telegram, note.com and WeChat posts without logging in

    • Remote
    • Freemium
    • Web search and scraping
  14. Read public Telegram channel posts with views, reactions and media, no login

    • Remote
    • Paid
    • Communication
  15. Track EU TikTok Ad Library ads by advertiser or keyword, with change status

    • Remote
    • Paid
    • Marketing and SEO
  16. Check which e-commerce stores publish a UCP profile and what it declares

    • Remote
    • Freemium
    • E-commerce
  17. YouTube transcripts, comments, video stats and channel monitoring for AI agents

    • 6 tools
    • Remote
    • Freemium
    • Design and media
  18. Search a free directory of MCP servers and x402 services, or list your own

    • 12 tools
    • Remote
    • Free
    • Web search and scraping
  19. Paid web, OpenAPI and Base chain evidence for agents, billed per call in USDC

    • 12 tools
    • Remote
    • Freemium
    • Web search and scraping