Skip to content

AI for web search and scraping

58 AI tools, agents, MCP servers and skills for web search and scraping, with pricing, platforms and alternatives for each.

Showing 1–30 of 58

  1. llms.ink

    MCP server

    Turn any website, GitHub repo, PDF or video link into llms.txt or llms-full.txt

    • 7 tools
    • Remote
    • Free
    • AI models and LLM tooling
  2. Search a catalogue of Apify Actors and n8n recipes for web-data jobs

    • 8 tools
    • Remote
    • Free
    • Web search and scraping
  3. Scraper Finder

    MCP server

    Look up the official API and ranked scraping tools for 172 websites

    • 3 tools
    • Remote
    • Free
    • Web search and scraping
  4. x-toolkit

    MCP server

    Read public X Articles, video transcripts and posts as plain text, no account needed

    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  5. CheckAnySite

    MCP server

    Ask an agent what a website is: AI builder traces, tech stack, traffic rank and trust

    • 6 tools
    • Remote
    • Free
    • Analytics and data
  6. Search a free directory of MCP servers and x402 services, or list your own

    • 12 tools
    • Remote
    • Free
    • Web search and scraping
  7. Free check of whether GPTBot, ClaudeBot and PerplexityBot can really reach a website

    • 1 tool
    • Remote
    • Free
    • Web search and scraping
  8. Test whether GPTBot, ClaudeBot, PerplexityBot and 10 more crawlers can reach a page

    • 1 tool
    • Remote
    • Free
    • Marketing and SEO
  9. Check whether a half-remembered thing can be found before your agent answers

    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  10. proxy-scraper

    MCP server

    Hourly-checked free HTTP and SOCKS proxy lists with a CLI and MCP tools to fetch pages

    Runs locally
    • Free
    • Web search and scraping
  11. Copy Markdown

    MCP server

    Turn any web page into clean Markdown, design tokens and a link list for LLMs

    Not checked yet
    • 4 tools
    • Remote
    • Free
    • Files and documents
  12. Vouch

    MCP server

    Check that links load and quotes really appear on a page, from your assistant

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Developer tools
  13. Find page-extraction services from a frozen catalogue; a time-limited free experiment

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Web search and scraping
  14. Agents Allowed

    MCP server

    Free lookup an agent can query to see if a site permits AI agents to browse, shop or book

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Web search and scraping
  15. Search immunology and virology literature, trials and grants across six databases at once

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Health and science
  16. Watch

    MCP server

    Watch a web page, CSS selector, JSON value or price and get changes by poll or webhook

    Not checked yet
    • 6 tools
    • Remote
    • Free
    • Web search and scraping
  17. Search expiring and recently dropped domain names by drop date, rank and linking domains

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Marketing and SEO
  18. Tracert

    MCP server

    Routes other agents mapped for web tasks, so yours verify instead of re-researching

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Web search and scraping
  19. Search 1,500+ indie-made products as alternatives to big-name software from your AI

    Not checked yet
    • 4 tools
    • Remote
    • Free
    • Web search and scraping
  20. Search a public catalogue of Telegram channels, groups, bots and listings, no key needed

    Not checked yet
    • 1 tool
    • Remote
    • Free
    • Communication
  21. Live news risk radars, market sensors and an AI anchor for agents over MCP

    Not checked yet
    • 12 tools
    • Remote
    • Free
    • Analytics and data
  22. MakerRadar

    MCP server

    Search launched products, alternatives and weekly launch boards on MakerRadar

    Not checked yet
    • 14 tools
    • Remote
    • Free
    • Web search and scraping
  23. Rivul

    MCP server

    Search 300M+ scholarly paper records from Claude, Cursor or Codex, free

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Health and science
  24. Check which AI crawlers a website allows in robots.txt, with the rule behind each verdict

    Not checked yet
    • 2 tools
    • Remote
    • Free
    • Web search and scraping
  25. WeProxy

    MCP server

    Turkey-focused proxy catalogue: product picks, pricing, coverage, error diagnosis

    Not checked yet
    • 7 tools
    • Remote
    • Free
    • Web search and scraping
  26. Nymwell Lookup

    MCP server

    Check domain availability and registrar prices and look up public domain records

    Not checked yet
    • 3 tools
    • Remote
    • Free
    • Web search and scraping
  27. X Reader MCP

    MCP server

    Read public X posts and full X Articles as clean Markdown, with no X API key

    Not checked yet
    • 1 tool
    • Remote
    • Free
    • Web search and scraping