Skip to content

MCP servers for web search and scraping

38 MCP servers that give AI live web search, page fetching and structured data extraction. Each one shows how to install it in Claude, Cursor and VS Code, whether it runs locally or remotely, and the tools it provides.
  1. Turn any website, GitHub repo, PDF or video link into llms.txt or llms-full.txt

    • 7 tools
    • Remote
    • Free
    • AI models and LLM tooling
  2. Read public X Articles, video transcripts and posts as plain text, no account needed

    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  3. Ask an agent what a website is: AI builder traces, tech stack, traffic rank and trust

    • 6 tools
    • Remote
    • Free
    • Analytics and data
  4. Search a free directory of MCP servers and x402 services, or list your own

    • 12 tools
    • Remote
    • Free
    • Web search and scraping
  5. Free check of whether GPTBot, ClaudeBot and PerplexityBot can really reach a website

    • 1 tool
    • Remote
    • Free
    • Web search and scraping
  6. Test whether GPTBot, ClaudeBot, PerplexityBot and 10 more crawlers can reach a page

    • 1 tool
    • Remote
    • Free
    • Marketing and SEO
  7. Check whether a half-remembered thing can be found before your agent answers

    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  8. Turn any web page into clean Markdown, design tokens and a link list for LLMs

    Not checked yet
    • 4 tools
    • Remote
    • Free
    • Files and documents
  9. Check that links load and quotes really appear on a page, from your assistant

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Developer tools
  10. Free lookup an agent can query to see if a site permits AI agents to browse, shop or book

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Web search and scraping
  11. Search immunology and virology literature, trials and grants across six databases at once

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Health and science
  12. Watch a web page, CSS selector, JSON value or price and get changes by poll or webhook

    Not checked yet
    • 6 tools
    • Remote
    • Free
    • Web search and scraping
  13. Search 1,500+ indie-made products as alternatives to big-name software from your AI

    Not checked yet
    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  14. Search a public catalogue of Telegram channels, groups, bots and listings, no key needed

    Not checked yet
    • 1 tool
    • Remote
    • Free
    • Communication
  15. Search 300M+ scholarly paper records from Claude, Cursor or Codex, free

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Health and science
  16. Turkey-focused proxy catalogue: product picks, pricing, coverage, error diagnosis

    Not checked yet
    • 7 tools
    • Remote
    • Free
    • Web search and scraping
  17. Check domain availability and registrar prices and look up public domain records

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Web search and scraping
  18. Read public X posts and full X Articles as clean Markdown, with no X API key

    Not checked yet
    • 1 tool
    • Remote
    • Free
    • Web search and scraping
  19. Look up conference and journal deadlines, CCF and CORE rankings and acceptance rates

    Not checked yet
    • 8 tools
    • Remote
    • Free
    • Web search and scraping
  20. Check if a site allows scraping, and get an extraction plan, before an agent fetches it

    Not checked yet
    • 8 tools
    • Remote
    • Free
    • Web search and scraping
  21. Dated residential proxy prices, terms and sourcing checks for 30 providers

    Not checked yet
    • 7 tools
    • Remote
    • Free
    • Web search and scraping
  22. Anonymous, no-key MCP server for SEO and AI crawler checks, robots.txt and llms.txt

    Not checked yet
    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  23. Query a 30-question benchmark of You.com, Tavily, Exa and Parallel search APIs

    Not checked yet
    • 12 tools
    • Remote
    • Free
    • Web search and scraping
  24. MCP search across Wikipedia, Crossref and community sources that keeps original citations

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Web search and scraping
  25. Public MCP server that searches Ithy articles and returns passages with links and sources

    Not checked yet
    • Remote
    • Free
    • Knowledge and memory
  26. Remote MCP to search, cite and verify research papers notarized on Soneium

    Not checked yet
    • Remote
    • Free
    • Knowledge and memory
  27. Search an interior-coordination service's website from your AI app, via Yomitasu

    Not checked yet
    • Remote
    • Free
    • Home services
  28. Check live domain availability, prices, brand scores and resale value from an assistant

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Web search and scraping
  29. Buy and manage ISP proxies from an AI client, with purchases that wait for your yes

    Not checked yet
    • 4 tools
    • Remote
    • Free
    • Web search and scraping

See every kind of AI for web search and scraping: tools, agents, MCP servers and skills.Ranked: the best AI for web search and scraping.