Skip to content
MCP server

robots.txt Allow count, body discarded

Remote MCP endpoint that reports a robots.txt Allow-directive count and discards the body

Install
Platforms
API
Runs
Remotely
Version
1.0.0
Listing updated
today

Similar MCP servers

About robots.txt Allow count, body discarded

robots.txt Allow count, body discarded is an MCP server for marketing and seo. It runs as a hosted remote server, so there is nothing to install locally.

This server is one entry in a collection of small, single-purpose public MCP endpoints published under the sadri-dridi registry namespace. Its registry description says it returns a count of Allow rules for robots.txt and discards the body afterwards, so the original text is not kept.

It is reachable only as a remote streamable-http endpoint hosted on a Cloudflare Workers domain; there is no installable package. The README groups it with sibling utilities for identifier layouts and calendar checks, and with separate doors that route agents to an advertising service. No tool names, parameters or output formats are documented for this particular server.

It suits autonomous clients that want a narrow robots.txt check without handling the full file themselves, though the available documentation is minimal.

Install

Run in a terminal

claude mcp add --transport http robots-allow-count https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp

Add to Settings → Connectors → Add custom connector

https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "robots-allow-count": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "robots-allow-count": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "robots-allow-count": {
      "serverUrl": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "robots-allow-count": {
      "type": "http",
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
    }
  }
}

Using several servers? Build one config file with the MCP config generator.

What it does

Allow directive count
Reports how many Allow rules appear in a robots.txt, per the registry description.
Body discarded
The robots.txt content is not retained after the count is produced.
Remote streamable-http
Connects over a hosted streamable-http MCP endpoint, with nothing to install.
Part of a utility set
Published alongside other small single-purpose MCP endpoints under the same registry namespace.

Good for

  • Check how many Allow rules a site's robots.txt declares from an agent workflow
  • Add a lightweight robots.txt signal to a crawl-readiness audit
  • Test remote MCP connectivity with a minimal single-purpose server

robots.txt Allow count, body discarded FAQ

How do I connect to it?

Use the remote streamable-http endpoint listed in the registry entry; no package is provided.

Does it keep the robots.txt content?

The registry description says the body is discarded.

Is it open source?

The repository is public on GitHub, but a licence is not stated in the available material.

How do I add robots.txt Allow count, body discarded to Claude Code?

Run this command in your terminal: claude mcp add --transport http robots-allow-count https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp

What are the best alternatives to robots.txt Allow count, body discarded?

The closest alternatives to robots.txt Allow count, body discarded are Sitemap URL count, body discarded and robots.txt group count.

Where this comes from

  • Description: written from the project's README and its MCP Registry entry.
  • Install steps: from the official MCP Registry entry.

How AIPoweredBot checks servers