Skip to content
MCP server

ActableSite AI Crawler Monitor

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt…

Install
Runs
Remotely
Version
1.4.2
Listing updated
4 months ago

Similar MCP servers

About ActableSite AI Crawler Monitor

ActableSite AI Crawler Monitor is an MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

Install

Run in a terminal

claude mcp add --transport http ai-crawler-robots-monitor https://actablesite.com/mcp/

Add to Settings → Connectors → Add custom connector

https://actablesite.com/mcp/
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "ai-crawler-robots-monitor": {
      "url": "https://actablesite.com/mcp/"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "ai-crawler-robots-monitor": {
      "type": "http",
      "url": "https://actablesite.com/mcp/"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "ai-crawler-robots-monitor": {
      "serverUrl": "https://actablesite.com/mcp/"
    }
  }
}

Add to Grok Bot desktop app → Connect apps → Add a custom connector

https://actablesite.com/mcp/

Enter a Name (for example "ActableSite AI Crawler Monitor"), choose Type: Remote HTTPS, paste this as the Server URL and choose Add plugin.

Full Grok Bot setup guide

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "ai-crawler-robots-monitor": {
      "type": "http",
      "url": "https://actablesite.com/mcp/"
    }
  }
}

Using several servers? Build one config file with the MCP config generator.

ActableSite AI Crawler Monitor FAQ

How do I add ActableSite AI Crawler Monitor to Claude Code?

Run this command in your terminal: claude mcp add --transport http ai-crawler-robots-monitor https://actablesite.com/mcp/

What are the best alternatives to ActableSite AI Crawler Monitor?

The closest alternatives to ActableSite AI Crawler Monitor are Crawlsight, Atalaya AI crawler check, Robots.txt & AI Crawler Checker, Site SEO & AI Crawler Audit and Ai Visibility.

Where this comes from

  • Description: from the official MCP Registry entry.
  • Install steps: from the official MCP Registry entry.

How AIPoweredBot checks servers