Agentwebmap
Find a site's llms.txt documentation, source links and check dates by domain or task
- Pricing
- Free
- Platforms
- Web, API
- Runs
- Remotely
- Version
- 0.8.2
- Listing updated
- 2 days ago
About Agentwebmap
Agentwebmap is a free MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.
Agent Web Map is a directory of llms.txt files with a JSON API and a hosted MCP server. Its aim is to let an agent find the publisher's own documentation for a domain or task, then see source links, matching evidence and when each file was last checked. The live site reports directory counts by category, led by ecommerce, marketing, local business and AI.
The MCP tools include resolve, search_index_preview, search_llms_registry, list_domains, lookup_domain, read_llms_txt and similar_sites, plus category, country and heading tools. read_llms_txt returns a cached copy with its source URL and dates, and an interactive 3D explorer links publishers that share headings. The site notes that shared headings are not an endorsement or proof of working capability.
Anonymous discovery covers 100 distinct publishers per UTC day across site, API and MCP; a free email-confirmed account with a named application raises that to 5,000 a day. Ordinary reads use stored data. The site also offers tools to audit a website, check an llms.txt file and generate a draft.
Install
Run in a terminal
claude mcp add --transport http agentwebmap https://agentwebmap.com/mcpAdd to Settings → Connectors → Add custom connector
https://agentwebmap.com/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"agentwebmap": {
"url": "https://agentwebmap.com/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"agentwebmap": {
"type": "http",
"url": "https://agentwebmap.com/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"agentwebmap": {
"serverUrl": "https://agentwebmap.com/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"agentwebmap": {
"type": "http",
"url": "https://agentwebmap.com/mcp"
}
}
}Tools this server provides (7)
Taken from the project documentation; the running server may differ.
- resolve
- Finds documentation for a task or domain.
- search_index_preview
- A small non-exhaustive keyword preview of the index.
- search_llms_registry
- Searches the llms.txt registry.
- list_domains
- Browses and filters publishers.
- lookup_domain
- Looks up a publisher by domain.
- read_llms_txt
- Reads a cached publisher llms.txt file.
- similar_sites
- Finds related publishers.
What it does
- Documentation lookup
- Resolves a domain or task to publisher documentation with evidence.
- Cached llms.txt reads
- Returns stored files with source URL and check dates.
- Related publishers
- Finds similar sites and shared headings.
- Free tiers
- Anonymous use plus higher daily limits with a free registered integration.
- Site audit tools
- Checks and generates llms.txt files for your own site.
Good for
- Find the official docs for MCP authorization
- Look up whether a domain publishes llms.txt
- Read a publisher's llms.txt with its last check date
- Discover sites with similar documentation sections
- Audit whether your own site is readable by agents
Pricing
Free
Free discovery; larger bulk access requires approval
Prices change. Check agentwebmap.com before you buy.
Agentwebmap FAQ
Do I need an account?
No. Basic discovery is free without an account or key.
What is the anonymous limit?
100 distinct publishers per UTC day across website, API and MCP.
How do I get more?
Register a free integration with a named application and declared purpose for 5,000 a day.
Do I need domain proof to read?
No. Domain ownership proof is for managing a listing, not for reading the directory.
Is Agentwebmap free?
Yes. Agentwebmap is free to use.
How do I add Agentwebmap to Claude Code?
Run this command in your terminal: claude mcp add --transport http agentwebmap https://agentwebmap.com/mcp
What are the best alternatives to Agentwebmap?
The closest alternatives to Agentwebmap are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.
Alternatives to Agentwebmap

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping

NeuralVerge
MCP serverStructured, source-tagged business and people data from 300+ sources for agents
- 200 tools
- Remote
- Free trial
- 1 GitHub star
- Web search and scraping

AI-Powered directory
MCP serverSearch AI tools, agents and MCP servers and get install configs from inside your AI assistant.
- 4 tools
- Remote
- Free
- Developer tools

BoardAgent
MCP serverRemote MCP server for a message board where only AI agents post, reply and vote
- 27 tools
- Remote
- Free
- Community directories