Octocrawl
Web scraping, mapping and crawling MCP server that returns clean Markdown
- Pricing
- Freemium
- Platforms
- API, CLI, self-hosted
- Runs
- Locally or Remotely
- Version
- 0.8.13
- Listing updated
- 3 days ago
About Octocrawl
Octocrawl is a freemium MCP server for web search and scraping. It can run locally on your machine or as a hosted remote server.
Octocrawl is reachable at a hosted URL with no account, where scrape and map work over plain HTTP within a daily allowance per address. Adding an account key spends monthly credits instead (500 a month on sign-up, 1 per HTTP page and 5 per browser-rendered page) and unlocks the browser lane, crawl and batch jobs, and an Amazon.sg product tool.
Every successful scrape comes with an Evidence Record describing how the page was captured, so an agent can check provenance before trusting extracted fields. Hosted crawls reach up to 500 pages and batches up to 1,000 URLs; larger jobs, page actions such as clicking and scrolling, and scheduled Monitors need the local API started with npx octocrawl serve and the @octocrawl/mcp package.
Output can be Markdown, HTML, links, image URLs and metadata, tables and listings as JSON or CSV, or fields matched to a JSON Schema. PDF text layers are read with page markers. The server can be restricted to read-only tools or specific tool groups through environment variables.
Install
Run in a terminal
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcpAdd to Settings → Connectors → Add custom connector
https://mcp.octocrawl.dev/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"octocrawl": {
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"octocrawl": {
"type": "http",
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"octocrawl": {
"serverUrl": "https://mcp.octocrawl.dev/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"octocrawl": {
"type": "http",
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}Configuration
W2L_API_URLThe Octocrawl API the server calls: npx octocrawl serve on this computer (default http://127.0.0.1:8787), or https://api.octocrawl.devW2L_API_TOKENA bearer token for a hosted or self-hosted API; none for a local oneW2L_MCP_TOOLSETSThe tool groups to list, comma-separated: core, crawl, batch, products, logins, monitor, delivery, or all (default all); the same as --toolsetsW2L_MCP_READ_ONLY1 lists read-only tools alone, with a scrape that does not act as the person; the same as --read-only
Tools this server provides (2)
Read directly from the server.
- scrape
- Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step.
- map
- List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the de
What it does
- Scrape and map
- Fetch page content as Markdown or list a site's URLs, filtered by search term.
- Crawl and batch jobs
- Follow links with depth and path limits, or submit lists of URLs.
- Evidence Record
- Each result records how the page was read.
- Structured extraction
- Pull tables and listings as JSON or CSV, or fields from a JSON Schema.
- Local Monitors
- Revisit sources on a schedule and deliver events to an HTTPS endpoint.
- Tool group controls
- Limit the server to chosen toolsets or read-only tools.
Good for
- Collect documentation into a knowledge base
- Compare product details across pages
- Map a site's URLs before a crawl
- Turn web tables into data for analysis
Pricing
Freemium
Keyless hosted use with a daily allowance; accounts get 500 credits a month
Prices change. Check octocrawl.dev before you buy.
Octocrawl FAQ
Do I need an API key?
No. Hosted scrape and map work keyless within a daily allowance; a key adds browser rendering, crawl and batch.
How do I run larger crawls?
Start the local API with npx octocrawl serve and connect it through the @octocrawl/mcp package.
Which clients are documented?
Claude Code, Cursor, OpenCode and Codex.
Can it read PDFs?
Yes, it extracts a PDF's text layer with page markers.
Is Octocrawl free?
Octocrawl has a free plan, with paid plans for more usage or features.
How do I add Octocrawl to Claude Code?
Run this command in your terminal: claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
What are the best alternatives to Octocrawl?
The closest alternatives to Octocrawl are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.
Alternatives to Octocrawl

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping

AI-Powered directory
MCP serverSearch AI tools, agents and MCP servers and get install configs from inside your AI assistant.
- 4 tools
- Remote
- Free
- Developer tools

Shopify MCP Server (any store)
MCP serverHosted MCP server that returns any Shopify store's products, prices and stock
Not checked yet- 4 tools
- Remote
- Paid
- E-commerce