# Agent Crawl Readiness

> Agent Crawl Readiness is an MCP server for web search and scraping. Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Website:** https://agent-ads-production-9f45.up.railway.app
- **Repository:** https://github.com/sadri-dridi/named-mcp-utilities/tree/HEAD/agent-ads/agent-crawl-readiness
- **MCP Registry name:** io.github.sadri-dridi/agent-crawl-readiness
- **Categories:** web search and scraping
- **Listing updated:** 2026-11-13

## About

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "agent-crawl-readiness": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "agent-crawl-readiness": {
      "serverUrl": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "agent-crawl-readiness": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}
```

## FAQ

### How do I add Agent Crawl Readiness to Claude Code?

Run this command in your terminal: claude mcp add --transport http agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

### What are the best alternatives to Agent Crawl Readiness?

The closest alternatives to Agent Crawl Readiness are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages.

## Where this comes from

- Description: from the official MCP Registry entry.
- Install steps: from the official MCP Registry entry.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages](https://ai-powered.com/mcp/onlinejobs-ph-scraper-apify-philippine-remote-and-va-jobs-wi): OnlineJobs.ph Scraper (Apify) - Philippine remote and VA jobs with wages is a paid MCP server for web search and scraping.
- [ImportYeti Scraper (Apify) - US import records and bills of lading](https://ai-powered.com/mcp/importyeti-scraper-apify-us-import-records-and-bills-of-ladi): ImportYeti Scraper (Apify) - US import records and bills of lading is a paid MCP server for web search and scraping.
- [Gupy Jobs Scraper (Apify) - Brazil jobs with descriptions](https://ai-powered.com/mcp/gupy-jobs-scraper-apify-brazil-jobs-with-descriptions): Gupy Jobs Scraper (Apify) - Brazil jobs with descriptions is a freemium MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/agent-crawl-readiness
