Skip to content

Agent skills for web search and scraping

10 agent skills that give AI live web search, page fetching and structured data extraction. Each one shows what it can run and how to install it in Claude Code, Codex, Cursor and other agents.
  1. Guides Qdrant monitoring setup including Prometheus scraping, health probes, Hybrid Cloud metrics, alerting, and log centralization

    • Open source
    • 41.3K GitHub stars
    • Knowledge and memory
  2. Build GitHub Copilot workflows with Xquik X API SDKs, REST endpoints, hosted Apify Actor runs, MCP tools, TweetClaw OpenClaw plugin installs

    • Open source
    • 41.3K GitHub stars
    • Web search and scraping
  3. Bing Webmaster Tools + IndexNow extension

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  4. Full-site crawling, scraping, and site mapping via Firecrawl MCP

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  5. Audit and improve content for AI Overviews and answer engines, including citability, entity clarity, crawler access, brand signals, and pass

    • Open source
    • 30.2K GitHub stars
    • Web search and scraping
  6. Use when the user wants to fact-check an article or verify factual claims in a document

    • Open source
    • 770 GitHub stars
    • Web search and scraping
  7. Generates Puppeteer scripts for browser automation, scraping, and PDF generation

    • Open source
    • 392 GitHub stars
    • Browser automation
  8. Guides Qdrant monitoring setup including Prometheus scraping, Datadog integration, health probes, Hybrid Cloud metrics, alerting, and log ce

    • Open source
    • 269 GitHub stars
    • Knowledge and memory
  9. Multi-level research: built-in knowledge → web search → Tavily deep research API

    • Open source
    • 107 GitHub stars
    • Web search and scraping
  10. Use Exa for semantic/neural web search

    • Open source
    • 72 GitHub stars
    • Web search and scraping

See every kind of AI for web search and scraping: tools, agents, MCP servers and skills.Ranked: the best AI for web search and scraping.