# Web Crawler MCP by InsightfulPipe

> Web Crawler MCP by InsightfulPipe is an MCP server for web search and scraping. Hosted MCP server with 33 technical SEO checks, from robots.txt to structured data. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Free trial
- **Starting price:** $29.99 a month
- **Platforms:** API
- **Website:** https://insightfulpipe.com/mcp-servers/crawler
- **Repository:** https://github.com/insightful-pipe/web-crawler-mcp-server
- **MCP Registry name:** com.insightfulpipe/crawler
- **Categories:** web search and scraping, marketing and SEO
- **Listing updated:** 2026-11-15

## About

Thirty-three read-only checkers cover on-page and technical SEO: meta tags, headings, canonicals, hreflang, indexability, internal links, image optimisation, structured data, Open Graph, security headers, SSL certificates and redirect chains. It also checks AI-search readiness with an AI crawler checker, a robots.txt checker and an llms.txt parser.

Beyond audits there are generators and extractors: an XML sitemap generator, a robots.txt generator with optional AI-bot restrictions, a link extractor and a brand extractor that pulls colours, fonts, buttons and logos from a site.

It is InsightfulPipe's own tool, so there is no third-party account to connect: sign in with InsightfulPipe and add the server URL. Actions can be switched on or off per account, and all 33 are read actions.

## Features

- **Technical SEO audit:** A comprehensive audit plus focused checks for meta, headings and canonicals.
- **AI crawler readiness:** Checks AI bot permissions, robots.txt and llms.txt.
- **Structured data checks:** Validates Schema.org markup and Open Graph metadata.
- **Performance and security:** Page speed, page load, SSL and security headers.
- **Sitemap and robots generators:** Produce an XML sitemap or robots.txt.
- **Brand extractor:** Reads colours, fonts, buttons and logos.

## Good for

- Audit a page for indexability and canonical issues
- Check which AI crawlers a site allows
- Validate hreflang and structured data
- Trace a redirect chain
- Generate a sitemap for a domain

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http crawler https://crawler.insightfulmcp.com/
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://crawler.insightfulmcp.com/
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "crawler": {
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "crawler": {
      "type": "http",
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "crawler": {
      "serverUrl": "https://crawler.insightfulmcp.com/"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "crawler": {
      "type": "http",
      "url": "https://crawler.insightfulmcp.com/"
    }
  }
}
```

## Tools

- `ai-crawler-checker`: Checks permissions for AI training bots and crawlers.
- `amp-checker`: Verifies AMP implementation.
- `brand-extractor`: Extracts colours, fonts, buttons and logos.
- `canonical-checker`: Validates canonical URLs.
- `headings-checker`: Reviews heading hierarchy.
- `hreflang-checker`: Reviews hreflang annotations.
- `indexability-checker`: Reviews robots and meta directives.
- `link-extractor`: Extracts and categorises links.
- `robots-txt-checker`: Parses robots.txt directives.
- `robots-txt-generator`: Generates robots.txt content.
- `sitemap-generator`: Crawls a domain and produces an XML sitemap.
- `structured-data-checker`: Validates Schema.org markup.
- `ssl-certificate-checker`: Inspects certificate chain and expiry.
- `status-code-checker`: Traces status and redirect chains.

## FAQ

### How many tools does it have?

33, all read actions.

### What does it cost?

It is included in InsightfulPipe plans from $29.99/month with a trial; the Crawler has a monthly usage limit per plan (5,000 on the entry tier).

### Do I need another account?

No, only an InsightfulPipe sign-in.

### Does it check AI search readiness?

Yes, with AI crawler, robots.txt and llms.txt checkers.

### Is Web Crawler MCP by InsightfulPipe free?

Web Crawler MCP by InsightfulPipe is a paid product with a free trial. Paid plans start at $29.99 a month.

### How do I add Web Crawler MCP by InsightfulPipe to Claude Code?

Run this command in your terminal: claude mcp add --transport http crawler https://crawler.insightfulmcp.com/

### What are the best alternatives to Web Crawler MCP by InsightfulPipe?

The closest alternatives to Web Crawler MCP by InsightfulPipe are Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP, SearXNG Search and Calven.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [Calven](https://ai-powered.com/mcp/calven): Calven is an MCP server for marketing and seo.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [MCP](https://ai-powered.com/mcp/mcp-mwhffe53): MCP is a freemium MCP server for marketing and seo.

---
Source: https://ai-powered.com/mcp/web-crawler-mcp-by-insightfulpipe
