Skip to content

AI for web search and scraping

230 AI tools, agents, MCP servers and skills for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–30 of 230

  1. Web search, scraping and page interaction API for AI agents and developers

    • Freemium
    • Agent framework
    • 202K GitHub stars
    • Web search and scraping
  2. Web crawler and hosted API that turns URLs into LLM-ready Markdown or JSON

    • Freemium
    • Open-source agent
    • 87.5K GitHub stars
    • Web search and scraping
  3. Open-source research agent that writes cited reports from the web and your documents

    • Open source
    • Open-source agent
    • 30.7K GitHub stars
    • Research
  4. Web API for agents: scrape, search, cloud browsers and task agents, billed in credits

    • Freemium
    • Open-source agent
    • 745 GitHub stars
    • Web search and scraping
  5. Self-hosted browser agent that scrapes protected sites from a plain-English prompt

    • Open source
    • Open-source agent
    • 180 GitHub stars
    • Web search and scraping
  6. ChangePing

    MCP server

    Watch web pages from Claude or ChatGPT and read summaries of what changed

    Not checked yet
    • 7 tools
    • Remote
    • Freemium
    • Web search and scraping
  7. ClawGrab

    MCP server

    Fetch transcripts of public YouTube, TikTok, Vimeo and podcast links for AI use

    Runs locally
    • 6 tools
    • Freemium
    • Web search and scraping
  8. llms.ink

    MCP server

    Turn any website, GitHub repo, PDF or video link into llms.txt or llms-full.txt

    • 7 tools
    • Remote
    • Free
    • AI models and LLM tooling
  9. x-toolkit

    MCP server

    Read public X Articles, video transcripts and posts as plain text, no account needed

    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  10. CheckAnySite

    MCP server

    Ask an agent what a website is: AI builder traces, tech stack, traffic rank and trust

    • 6 tools
    • Remote
    • Free
    • Analytics and data
  11. Turn YouTube captions, PDFs and web pages into clean Markdown for RAG

    • Remote
    • Freemium
    • Knowledge and memory
  12. Read Bluesky, Telegram, note.com and WeChat posts without logging in

    • Remote
    • Freemium
    • Web search and scraping
  13. Check which e-commerce stores publish a UCP profile and what it declares

    • Remote
    • Freemium
    • E-commerce
  14. YouTube transcripts, comments, video stats and channel monitoring for AI agents

    • 6 tools
    • Remote
    • Freemium
    • Design and media
  15. Search a free directory of MCP servers and x402 services, or list your own

    • 12 tools
    • Remote
    • Free
    • Web search and scraping
  16. Paid web, OpenAPI and Base chain evidence for agents, billed per call in USDC

    • 12 tools
    • Remote
    • Freemium
    • Web search and scraping
  17. Browserless

    MCP server

    Give an assistant a cloud browser to scrape, search, crawl and audit pages

    • 5 tools
    • Remote
    • Freemium
    • Browser automation
  18. ResearchZosho

    MCP server

    Local deep research assistant that checks each citation and keeps what it finds

    Runs locally
    • Open source
    • Knowledge and memory
  19. Free check of whether GPTBot, ClaudeBot and PerplexityBot can really reach a website

    • 1 tool
    • Remote
    • Free
    • Web search and scraping
  20. Search what was said on YouTube and podcasts and get quotes cued to the second

    • 4 tools
    • Remote
    • Freemium
    • Design and media
  21. SpecStream

    MCP server

    Web search, page extraction and structured live-web datasets for agents

    • 3 tools
    • Remote
    • Freemium
    • Web search and scraping
  22. Site-Shot

    MCP server

    Website screenshots for AI agents: full page, ad and cookie-banner removal

    • Remote
    • Freemium
    • Web search and scraping
  23. QuietDiff

    MCP server

    Watch chosen fields on web pages and have an assistant create monitors and read changes

    • 9 tools
    • Remote
    • Freemium
    • Web search and scraping
  24. FetchLayer

    MCP server

    Live data from Reddit, X, TikTok, app stores, ad libraries and 30+ more platforms

    • Remote
    • Freemium
    • Web search and scraping
  25. Konexium

    MCP server

    Research African companies from an assistant, with cited answers and spend approval

    Not checked yet
    • 2 tools
    • Remote
    • Freemium
    • Sales and CRM
  26. Test whether GPTBot, ClaudeBot, PerplexityBot and 10 more crawlers can reach a page

    • 1 tool
    • Remote
    • Free
    • Marketing and SEO