Skip to content
MCP server

Web Crawler MCP by InsightfulPipe

Hosted MCP server with 33 technical SEO checks, from robots.txt to structured data

Install
Pricing
Free trial
Platforms
API
Runs
Remotely
Version
1.0.0
Listing updated
2 days ago

About Web Crawler MCP by InsightfulPipe

Web Crawler MCP by InsightfulPipe is an MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Thirty-three read-only checkers cover on-page and technical SEO: meta tags, headings, canonicals, hreflang, indexability, internal links, image optimisation, structured data, Open Graph, security headers, SSL certificates and redirect chains. It also checks AI-search readiness with an AI crawler checker, a robots.txt checker and an llms.txt parser.

Beyond audits there are generators and extractors: an XML sitemap generator, a robots.txt generator with optional AI-bot restrictions, a link extractor and a brand extractor that pulls colours, fonts, buttons and logos from a site.

It is InsightfulPipe's own tool, so there is no third-party account to connect: sign in with InsightfulPipe and add the server URL. Actions can be switched on or off per account, and all 33 are read actions.

Install

Run in a terminal

claude mcp add --transport http crawler https://crawler.insightfulmcp.com/

Add to Settings → Connectors → Add custom connector

https://crawler.insightfulmcp.com/
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "crawler": {
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "crawler": {
      "type": "http",
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "crawler": {
      "serverUrl": "https://crawler.insightfulmcp.com/"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "crawler": {
      "type": "http",
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}

Tools this server provides (14)

Taken from the project documentation; the running server may differ.

ai-crawler-checker
Checks permissions for AI training bots and crawlers.
amp-checker
Verifies AMP implementation.
brand-extractor
Extracts colours, fonts, buttons and logos.
canonical-checker
Validates canonical URLs.
headings-checker
Reviews heading hierarchy.
hreflang-checker
Reviews hreflang annotations.
indexability-checker
Reviews robots and meta directives.
link-extractor
Extracts and categorises links.
robots-txt-checker
Parses robots.txt directives.
robots-txt-generator
Generates robots.txt content.
sitemap-generator
Crawls a domain and produces an XML sitemap.
structured-data-checker
Validates Schema.org markup.
ssl-certificate-checker
Inspects certificate chain and expiry.
status-code-checker
Traces status and redirect chains.

What it does

Technical SEO audit
A comprehensive audit plus focused checks for meta, headings and canonicals.
AI crawler readiness
Checks AI bot permissions, robots.txt and llms.txt.
Structured data checks
Validates Schema.org markup and Open Graph metadata.
Performance and security
Page speed, page load, SSL and security headers.
Sitemap and robots generators
Produce an XML sitemap or robots.txt.
Brand extractor
Reads colours, fonts, buttons and logos.

Good for

  • Audit a page for indexability and canonical issues
  • Check which AI crawlers a site allows
  • Validate hreflang and structured data
  • Trace a redirect chain
  • Generate a sitemap for a domain

Pricing

Free trial, from $29.99 a month

Included in InsightfulPipe plans from $29.99/month; trial available

Prices change. Check insightfulpipe.com before you buy.

Web Crawler MCP by InsightfulPipe FAQ

How many tools does it have?

33, all read actions.

What does it cost?

It is included in InsightfulPipe plans from $29.99/month with a trial; the Crawler has a monthly usage limit per plan (5,000 on the entry tier).

Do I need another account?

No, only an InsightfulPipe sign-in.

Does it check AI search readiness?

Yes, with AI crawler, robots.txt and llms.txt checkers.

Is Web Crawler MCP by InsightfulPipe free?

Web Crawler MCP by InsightfulPipe is a paid product with a free trial. Paid plans start at $29.99 a month.

How do I add Web Crawler MCP by InsightfulPipe to Claude Code?

Run this command in your terminal: claude mcp add --transport http crawler https://crawler.insightfulmcp.com/

What are the best alternatives to Web Crawler MCP by InsightfulPipe?

The closest alternatives to Web Crawler MCP by InsightfulPipe are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

Alternatives to Web Crawler MCP by InsightfulPipe

  1. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  2. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  3. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  4. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  5. Calven

    MCP server

    Product marketing context from Calven delivered into Claude, ChatGPT and other AI tools

    Not checked yet
    • Remote
    • Free trial
    • 3 GitHub stars
    • Marketing and SEO
  6. NeuralVerge

    MCP server

    Structured, source-tagged business and people data from 300+ sources for agents

    • 200 tools
    • Remote
    • Free trial
    • 1 GitHub star
    • Web search and scraping