# robots.txt Allow count, body discarded

> robots.txt Allow count, body discarded is an MCP server for marketing and seo. Remote MCP endpoint that reports a robots.txt Allow-directive count and discards the body. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Platforms:** API
- **Repository:** https://github.com/sadri-dridi/named-mcp-utilities
- **MCP Registry name:** io.github.sadri-dridi/robots-allow-count
- **Categories:** [marketing and SEO](https://ai-powered.com/categories/marketing-and-seo)
- **Listing updated:** 2026-11-18

## About

This server is one entry in a collection of small, single-purpose public MCP endpoints published under the sadri-dridi registry namespace. Its registry description says it returns a count of Allow rules for robots.txt and discards the body afterwards, so the original text is not kept.

It is reachable only as a remote streamable-http endpoint hosted on a Cloudflare Workers domain; there is no installable package. The README groups it with sibling utilities for identifier layouts and calendar checks, and with separate doors that route agents to an advertising service. No tool names, parameters or output formats are documented for this particular server.

It suits autonomous clients that want a narrow robots.txt check without handling the full file themselves, though the available documentation is minimal.

## Features

- **Allow directive count:** Reports how many Allow rules appear in a robots.txt, per the registry description.
- **Body discarded:** The robots.txt content is not retained after the count is produced.
- **Remote streamable-http:** Connects over a hosted streamable-http MCP endpoint, with nothing to install.
- **Part of a utility set:** Published alongside other small single-purpose MCP endpoints under the same registry namespace.

## Good for

- Check how many Allow rules a site's robots.txt declares from an agent workflow
- Add a lightweight robots.txt signal to a crawl-readiness audit
- Test remote MCP connectivity with a minimal single-purpose server

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http robots-allow-count https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "robots-allow-count": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "robots-allow-count": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "robots-allow-count": {
      "serverUrl": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "robots-allow-count": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}
```

## FAQ

### How do I connect to it?

Use the remote streamable-http endpoint listed in the registry entry; no package is provided.

### Does it keep the robots.txt content?

The registry description says the body is discarded.

### Is it open source?

The repository is public on GitHub, but a licence is not stated in the available material.

### How do I add robots.txt Allow count, body discarded to Claude Code?

Run this command in your terminal: claude mcp add --transport http robots-allow-count https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp

### What are the best alternatives to robots.txt Allow count, body discarded?

The closest alternatives to robots.txt Allow count, body discarded are Sitemap URL count, body discarded and robots.txt group count.

## Where this comes from

- Description: written from the project's README and its MCP Registry entry.
- Install steps: from the official MCP Registry entry.

## Alternatives

- [Sitemap URL count, body discarded](https://ai-powered.com/mcp/sitemap-url-count-body-discarded): Sitemap URL count, body discarded is an MCP server for web search and scraping.
- [robots.txt group count](https://ai-powered.com/mcp/robots-txt-group-count): robots.txt group count is an MCP server for web search and scraping.

---
Source: https://ai-powered.com/mcp/robots-txt-allow-count-body-discarded
