Skip to content
MCP server

WebCrawlerAPI

Hosted MCP server that turns any public web page into clean Markdown for an assistant

Install
Pricing
Paid
Platforms
API, Web
Runs
Remotely
Version
0.1.3
Listing updated
3 days ago

About WebCrawlerAPI

WebCrawlerAPI is a paid MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

A single read-only tool, get_markdown, fetches a public page and returns its main content as Markdown. JavaScript-rendered pages work, and navigation, ads and boilerplate are stripped. The input is one URL of up to 2048 characters.

The server is hosted, so there is nothing to install. You add its URL to Claude Code, Claude.ai or Desktop, Cursor, VS Code or ChatGPT and sign in with a WebCrawlerAPI account through OAuth 2.1; no API key is pasted. Tokens are tied to the server URL and to the organization you approve, and every call is billed to that organization.

Failures come back as short messages, for example an empty balance, a page blocked by robots.txt, an unreachable host, a protected page or one that is too large. Rate limits are 120 requests a minute per organization and 300 per IP, with request bodies capped at 256 KB.

Install

Run in a terminal

claude mcp add --transport http mcp-server https://mcp.webcrawlerapi.com/mcp

Add to Settings → Connectors → Add custom connector

https://mcp.webcrawlerapi.com/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "mcp-server": {
      "url": "https://mcp.webcrawlerapi.com/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "mcp-server": {
      "type": "http",
      "url": "https://mcp.webcrawlerapi.com/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "mcp-server": {
      "serverUrl": "https://mcp.webcrawlerapi.com/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "mcp-server": {
      "type": "http",
      "url": "https://mcp.webcrawlerapi.com/mcp"
    }
  }
}

Tools this server provides (1)

Taken from the project documentation; the running server may differ.

get_markdown
Fetches a public web page and returns its main content as Markdown.

What it does

Page to Markdown
Reads any public http or https page and returns its main content.
JavaScript rendering
Pages that render client-side are supported.
OAuth sign-in
Authenticate with your account; no API key to paste.
Clear error messages
Failures such as blocked pages or empty balance return short explanations.
Read-only tool
The tool is marked read-only and idempotent.
Many clients
Setup steps for Claude, Cursor, VS Code, ChatGPT and MCP Inspector.

Good for

  • Pull documentation pages into a chat for summarising
  • Gather article text for a knowledge base
  • Read product pages without ads and navigation noise
  • Let an assistant check a live page before answering

Pricing

Paid, from $99 a month

Pay per page from $0.002; Professional $99/mo, Business $499/mo

Prices change. Check webcrawlerapi.com before you buy.

WebCrawlerAPI FAQ

Do I need an API key?

No. Sign-in is OAuth; you approve access for one of your organizations.

What does a call cost?

Calls are billed to your organization. The pricing page lists pay as you go from $0.002 a page and $99 a month for Professional.

What are the limits?

120 requests per minute per organization, 300 per minute per IP, and 256 KB request bodies.

Which pages can it read?

Public http or https pages. Sites that block the request or robots.txt return an error.

Is WebCrawlerAPI free?

No. WebCrawlerAPI is a paid product. Paid plans start at $99 a month.

How do I add WebCrawlerAPI to Claude Code?

Run this command in your terminal: claude mcp add --transport http mcp-server https://mcp.webcrawlerapi.com/mcp

What are the best alternatives to WebCrawlerAPI?

The closest alternatives to WebCrawlerAPI are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.

Alternatives to WebCrawlerAPI

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  4. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping
  5. Search AI tools, agents and MCP servers and get install configs from inside your AI assistant.

    • 4 tools
    • Remote
    • Free
    • Developer tools
  6. Hosted MCP server that returns any Shopify store's products, prices and stock

    Not checked yet
    • 4 tools
    • Remote
    • Paid
    • E-commerce