Skip to content

MCP servers for web search and scraping

6 MCP servers that give AI live web search, page fetching and structured data extraction. Each one shows how to install it in Claude, Cursor and VS Code, whether it runs locally or remotely, and the tools it provides.
  1. Fetch transcripts of public YouTube, TikTok, Vimeo and podcast links for AI use

    Runs locally
    • 6 tools
    • Freemium
    • Web search and scraping
  2. Live Google results and domain rank checks by country and device, pay per search

    Runs locally
    • 3 tools
    • Freemium
    • Web search and scraping
  3. Scrape, crawl, search and extract web pages as clean Markdown or JSON for agents

    Not checked yet
    • Remote
    • Freemium
    • Web search and scraping
  4. Check which AI crawlers can read a site and generate llms.txt, robots.txt and schema

    Runs locally
    • 9 tools
    • Freemium
    • Web search and scraping
  5. Ask Claude about Reddit and Hacker News threads you exported with Comment Scraper

    Runs locally
    • 4 tools
    • Freemium
    • Web search and scraping

See every kind of AI for web search and scraping: tools, agents, MCP servers and skills.Ranked: the best AI for web search and scraping.