Skip to content
MCP server

Octocrawl

Web scraping, mapping and crawling MCP server that returns clean Markdown

Pricing
Freemium
Platforms
API, CLI, self-hosted
Runs
Locally or Remotely
Version
0.8.13
Listing updated
3 days ago

About Octocrawl

Octocrawl is a freemium MCP server for web search and scraping. It can run locally on your machine or as a hosted remote server.

Octocrawl is reachable at a hosted URL with no account, where scrape and map work over plain HTTP within a daily allowance per address. Adding an account key spends monthly credits instead (500 a month on sign-up, 1 per HTTP page and 5 per browser-rendered page) and unlocks the browser lane, crawl and batch jobs, and an Amazon.sg product tool.

Every successful scrape comes with an Evidence Record describing how the page was captured, so an agent can check provenance before trusting extracted fields. Hosted crawls reach up to 500 pages and batches up to 1,000 URLs; larger jobs, page actions such as clicking and scrolling, and scheduled Monitors need the local API started with npx octocrawl serve and the @octocrawl/mcp package.

Output can be Markdown, HTML, links, image URLs and metadata, tables and listings as JSON or CSV, or fields matched to a JSON Schema. PDF text layers are read with page markers. The server can be restricted to read-only tools or specific tool groups through environment variables.

Install

Run in a terminal

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

Add to Settings → Connectors → Add custom connector

https://mcp.octocrawl.dev/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "octocrawl": {
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "octocrawl": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "octocrawl": {
      "serverUrl": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "octocrawl": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}

Configuration

  • W2L_API_URLThe Octocrawl API the server calls: npx octocrawl serve on this computer (default http://127.0.0.1:8787), or https://api.octocrawl.dev
  • W2L_API_TOKENA bearer token for a hosted or self-hosted API; none for a local one
  • W2L_MCP_TOOLSETSThe tool groups to list, comma-separated: core, crawl, batch, products, logins, monitor, delivery, or all (default all); the same as --toolsets
  • W2L_MCP_READ_ONLY1 lists read-only tools alone, with a scrape that does not act as the person; the same as --read-only

Tools this server provides (2)

Read directly from the server.

scrape
Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step.
map
List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the de

What it does

Scrape and map
Fetch page content as Markdown or list a site's URLs, filtered by search term.
Crawl and batch jobs
Follow links with depth and path limits, or submit lists of URLs.
Evidence Record
Each result records how the page was read.
Structured extraction
Pull tables and listings as JSON or CSV, or fields from a JSON Schema.
Local Monitors
Revisit sources on a schedule and deliver events to an HTTPS endpoint.
Tool group controls
Limit the server to chosen toolsets or read-only tools.

Good for

  • Collect documentation into a knowledge base
  • Compare product details across pages
  • Map a site's URLs before a crawl
  • Turn web tables into data for analysis

Pricing

Freemium

Keyless hosted use with a daily allowance; accounts get 500 credits a month

Prices change. Check octocrawl.dev before you buy.

Octocrawl FAQ

Do I need an API key?

No. Hosted scrape and map work keyless within a daily allowance; a key adds browser rendering, crawl and batch.

How do I run larger crawls?

Start the local API with npx octocrawl serve and connect it through the @octocrawl/mcp package.

Which clients are documented?

Claude Code, Cursor, OpenCode and Codex.

Can it read PDFs?

Yes, it extracts a PDF's text layer with page markers.

Is Octocrawl free?

Octocrawl has a free plan, with paid plans for more usage or features.

How do I add Octocrawl to Claude Code?

Run this command in your terminal: claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

What are the best alternatives to Octocrawl?

The closest alternatives to Octocrawl are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.

Alternatives to Octocrawl

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  4. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  5. Search AI tools, agents and MCP servers and get install configs from inside your AI assistant.

    • 4 tools
    • Remote
    • Free
    • Developer tools
  6. Hosted MCP server that returns any Shopify store's products, prices and stock

    Not checked yet
    • 4 tools
    • Remote
    • Paid
    • E-commerce