robots.txt Allow count, body discarded
Remote MCP endpoint that reports a robots.txt Allow-directive count and discards the body
- Platforms
- API
- Runs
- Remotely
- Version
- 1.0.0
- Listing updated
- today
Similar MCP servers
About robots.txt Allow count, body discarded
robots.txt Allow count, body discarded is an MCP server for marketing and seo. It runs as a hosted remote server, so there is nothing to install locally.
This server is one entry in a collection of small, single-purpose public MCP endpoints published under the sadri-dridi registry namespace. Its registry description says it returns a count of Allow rules for robots.txt and discards the body afterwards, so the original text is not kept.
It is reachable only as a remote streamable-http endpoint hosted on a Cloudflare Workers domain; there is no installable package. The README groups it with sibling utilities for identifier layouts and calendar checks, and with separate doors that route agents to an advertising service. No tool names, parameters or output formats are documented for this particular server.
It suits autonomous clients that want a narrow robots.txt check without handling the full file themselves, though the available documentation is minimal.
Install
Run in a terminal
claude mcp add --transport http robots-allow-count https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcpAdd to Settings → Connectors → Add custom connector
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"robots-allow-count": {
"url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"robots-allow-count": {
"type": "http",
"url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"robots-allow-count": {
"serverUrl": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"robots-allow-count": {
"type": "http",
"url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp"
}
}
}Using several servers? Build one config file with the MCP config generator.
What it does
- Allow directive count
- Reports how many Allow rules appear in a robots.txt, per the registry description.
- Body discarded
- The robots.txt content is not retained after the count is produced.
- Remote streamable-http
- Connects over a hosted streamable-http MCP endpoint, with nothing to install.
- Part of a utility set
- Published alongside other small single-purpose MCP endpoints under the same registry namespace.
Good for
- Check how many Allow rules a site's robots.txt declares from an agent workflow
- Add a lightweight robots.txt signal to a crawl-readiness audit
- Test remote MCP connectivity with a minimal single-purpose server
robots.txt Allow count, body discarded FAQ
How do I connect to it?
Use the remote streamable-http endpoint listed in the registry entry; no package is provided.
Does it keep the robots.txt content?
The registry description says the body is discarded.
Is it open source?
The repository is public on GitHub, but a licence is not stated in the available material.
How do I add robots.txt Allow count, body discarded to Claude Code?
Run this command in your terminal: claude mcp add --transport http robots-allow-count https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-allow-count/mcp
What are the best alternatives to robots.txt Allow count, body discarded?
The closest alternatives to robots.txt Allow count, body discarded are Sitemap URL count, body discarded and robots.txt group count.
Where this comes from
- Description: written from the project's README and its MCP Registry entry.
- Install steps: from the official MCP Registry entry.