Skip to content

AI for web search and scraping

10 AI tools, agents, MCP servers and skills for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–10 of 10

  1. Guides Qdrant monitoring setup including Prometheus scraping, health probes, Hybrid Cloud metrics, alerting, and log centralization

    • Open source
    • 41.3K GitHub stars
    • Knowledge and memory
  2. Build GitHub Copilot workflows with Xquik X API SDKs, REST endpoints, hosted Apify Actor runs, MCP tools, TweetClaw OpenClaw plugin installs

    • Open source
    • 41.3K GitHub stars
    • Web search and scraping
  3. Bing Webmaster Tools + IndexNow extension

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  4. Full-site crawling, scraping, and site mapping via Firecrawl MCP

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  5. SEO geo

    Skill

    Audit and improve content for AI Overviews and answer engines, including citability, entity clarity, crawler access, brand signals, and pass

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  6. Use when the user wants to fact-check an article or verify factual claims in a document

    • Open source
    • 770 GitHub stars
    • Web search and scraping
  7. Generates Puppeteer scripts for browser automation, scraping, and PDF generation

    • Open source
    • 392 GitHub stars
    • Browser automation
  8. Guides Qdrant monitoring setup including Prometheus scraping, Datadog integration, health probes, Hybrid Cloud metrics, alerting, and log ce

    • Open source
    • 269 GitHub stars
    • Knowledge and memory
  9. Multi-level research: built-in knowledge → web search → Tavily deep research API

    • Open source
    • 107 GitHub stars
    • Web search and scraping
  10. Use Exa for semantic/neural web search

    • Open source
    • 72 GitHub stars
    • Web search and scraping