# Octocrawl

> Octocrawl is a freemium MCP server for web search and scraping. Web scraping, mapping and crawling MCP server that returns clean Markdown. It can run locally on your machine or as a hosted remote server.

- **Kind:** MCP server
- **Pricing:** Freemium
- **Platforms:** API, CLI, self-hosted
- **Website:** https://octocrawl.dev/docs/connect-mcp/
- **Repository:** https://github.com/77777r7/octocrawl
- **MCP Registry name:** io.github.77777R7/octocrawl
- **Categories:** web search and scraping
- **Listing updated:** 2026-11-14

## About

Octocrawl is reachable at a hosted URL with no account, where scrape and map work over plain HTTP within a daily allowance per address. Adding an account key spends monthly credits instead (500 a month on sign-up, 1 per HTTP page and 5 per browser-rendered page) and unlocks the browser lane, crawl and batch jobs, and an Amazon.sg product tool.

Every successful scrape comes with an Evidence Record describing how the page was captured, so an agent can check provenance before trusting extracted fields. Hosted crawls reach up to 500 pages and batches up to 1,000 URLs; larger jobs, page actions such as clicking and scrolling, and scheduled Monitors need the local API started with npx octocrawl serve and the @octocrawl/mcp package.

Output can be Markdown, HTML, links, image URLs and metadata, tables and listings as JSON or CSV, or fields matched to a JSON Schema. PDF text layers are read with page markers. The server can be restricted to read-only tools or specific tool groups through environment variables.

## Features

- **Scrape and map:** Fetch page content as Markdown or list a site's URLs, filtered by search term.
- **Crawl and batch jobs:** Follow links with depth and path limits, or submit lists of URLs.
- **Evidence Record:** Each result records how the page was read.
- **Structured extraction:** Pull tables and listings as JSON or CSV, or fields from a JSON Schema.
- **Local Monitors:** Revisit sources on a schedule and deliver events to an HTTPS endpoint.
- **Tool group controls:** Limit the server to chosen toolsets or read-only tools.

## Good for

- Collect documentation into a knowledge base
- Compare product details across pages
- Map a site's URLs before a crawl
- Turn web tables into data for analysis

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://mcp.octocrawl.dev/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "octocrawl": {
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "octocrawl": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "octocrawl": {
      "serverUrl": "https://mcp.octocrawl.dev/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "octocrawl": {
      "type": "http",
      "url": "https://mcp.octocrawl.dev/mcp"
    }
  }
}
```

## Tools

- `scrape`: Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step.
- `map`: List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the de

## FAQ

### Do I need an API key?

No. Hosted scrape and map work keyless within a daily allowance; a key adds browser rendering, crawl and batch.

### How do I run larger crawls?

Start the local API with npx octocrawl serve and connect it through the @octocrawl/mcp package.

### Which clients are documented?

Claude Code, Cursor, OpenCode and Codex.

### Can it read PDFs?

Yes, it extracts a PDF's text layer with page markers.

### Is Octocrawl free?

Octocrawl has a free plan, with paid plans for more usage or features.

### How do I add Octocrawl to Claude Code?

Run this command in your terminal: claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

### What are the best alternatives to Octocrawl?

The closest alternatives to Octocrawl are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [Shopify MCP Server (any store)](https://ai-powered.com/mcp/shopify-mcp-server-any-store): Shopify MCP Server (any store) is a paid MCP server for e-commerce.
- [Chanroll: YouTube Channel Video Scraper (Official API)](https://ai-powered.com/mcp/chanroll-youtube-channel-video-scraper-official-api): Chanroll: YouTube Channel Video Scraper (Official API) is a paid MCP server for design and media.
- [Vidvett: YouTube Video Details Scraper (Official API)](https://ai-powered.com/mcp/vidvett-youtube-video-details-scraper-official-api): Vidvett: YouTube Video Details Scraper (Official API) is a paid MCP server for design and media.

---
Source: https://ai-powered.com/mcp/octocrawl
