
Web search, scraping and page interaction API for AI agents and developers
- Freemium
- Agent framework
- 202K GitHub stars
- Web search and scraping

Web search, scraping and page interaction API for AI agents and developers

Web crawler and hosted API that turns URLs into LLM-ready Markdown or JSON

Open-source research agent that writes cited reports from the web and your documents

Web API for agents: scrape, search, cloud browsers and task agents, billed in credits

Self-hosted browser agent that scrapes protected sites from a plain-English prompt
Watch web pages from Claude or ChatGPT and read summaries of what changed
Not checked yetFetch transcripts of public YouTube, TikTok, Vimeo and podcast links for AI use
Runs locallyTurn any website, GitHub repo, PDF or video link into llms.txt or llms-full.txt
Read public X Articles, video transcripts and posts as plain text, no account needed
Ask an agent what a website is: AI builder traces, tech stack, traffic rank and trust
Turn YouTube captions, PDFs and web pages into clean Markdown for RAG
Read Bluesky, Telegram, note.com and WeChat posts without logging in
Check which e-commerce stores publish a UCP profile and what it declares
YouTube transcripts, comments, video stats and channel monitoring for AI agents
Search HiringCafe jobs by title, place and pay, with employer apply links
Search a free directory of MCP servers and x402 services, or list your own
Paid web, OpenAPI and Base chain evidence for agents, billed per call in USDC
Search every Ashby job board at once, newest first, with optional salary and details
Search every Cornerstone OnDemand career site at once, newest jobs first
Search every Greenhouse job board at once, with salary ranges and posting dates
Give an assistant a cloud browser to scrape, search, crawl and audit pages
Local deep research assistant that checks each citation and keeps what it finds
Runs locallyFree check of whether GPTBot, ClaudeBot and PerplexityBot can really reach a website
Search what was said on YouTube and podcasts and get quotes cued to the second
Web search, page extraction and structured live-web datasets for agents
Website screenshots for AI agents: full page, ad and cookie-banner removal
Watch chosen fields on web pages and have an assistant create monitors and read changes
Live data from Reddit, X, TikTok, app stores, ad libraries and 30+ more platforms
Research African companies from an assistant, with cited answers and spend approval
Not checked yetTest whether GPTBot, ClaudeBot, PerplexityBot and 10 more crawlers can reach a page