# Agentwebmap

> Agentwebmap is a free MCP server for web search and scraping. Find a site's llms.txt documentation, source links and check dates by domain or task. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Free
- **Platforms:** Web, API
- **Website:** https://agentwebmap.com
- **Repository:** https://github.com/agentwebmap/mcp
- **MCP Registry name:** com.agentwebmap/agentwebmap
- **Categories:** web search and scraping, community directories
- **Listing updated:** 2026-11-14

## About

Agent Web Map is a directory of llms.txt files with a JSON API and a hosted MCP server. Its aim is to let an agent find the publisher's own documentation for a domain or task, then see source links, matching evidence and when each file was last checked. The live site reports directory counts by category, led by ecommerce, marketing, local business and AI.

The MCP tools include resolve, search_index_preview, search_llms_registry, list_domains, lookup_domain, read_llms_txt and similar_sites, plus category, country and heading tools. read_llms_txt returns a cached copy with its source URL and dates, and an interactive 3D explorer links publishers that share headings. The site notes that shared headings are not an endorsement or proof of working capability.

Anonymous discovery covers 100 distinct publishers per UTC day across site, API and MCP; a free email-confirmed account with a named application raises that to 5,000 a day. Ordinary reads use stored data. The site also offers tools to audit a website, check an llms.txt file and generate a draft.

## Features

- **Documentation lookup:** Resolves a domain or task to publisher documentation with evidence.
- **Cached llms.txt reads:** Returns stored files with source URL and check dates.
- **Related publishers:** Finds similar sites and shared headings.
- **Free tiers:** Anonymous use plus higher daily limits with a free registered integration.
- **Site audit tools:** Checks and generates llms.txt files for your own site.

## Good for

- Find the official docs for MCP authorization
- Look up whether a domain publishes llms.txt
- Read a publisher's llms.txt with its last check date
- Discover sites with similar documentation sections
- Audit whether your own site is readable by agents

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http agentwebmap https://agentwebmap.com/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://agentwebmap.com/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "agentwebmap": {
      "url": "https://agentwebmap.com/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "agentwebmap": {
      "type": "http",
      "url": "https://agentwebmap.com/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "agentwebmap": {
      "serverUrl": "https://agentwebmap.com/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "agentwebmap": {
      "type": "http",
      "url": "https://agentwebmap.com/mcp"
    }
  }
}
```

## Tools

- `resolve`: Finds documentation for a task or domain.
- `search_index_preview`: A small non-exhaustive keyword preview of the index.
- `search_llms_registry`: Searches the llms.txt registry.
- `list_domains`: Browses and filters publishers.
- `lookup_domain`: Looks up a publisher by domain.
- `read_llms_txt`: Reads a cached publisher llms.txt file.
- `similar_sites`: Finds related publishers.

## FAQ

### Do I need an account?

No. Basic discovery is free without an account or key.

### What is the anonymous limit?

100 distinct publishers per UTC day across website, API and MCP.

### How do I get more?

Register a free integration with a named application and declared purpose for 5,000 a day.

### Do I need domain proof to read?

No. Domain ownership proof is for managing a listing, not for reading the directory.

### Is Agentwebmap free?

Yes. Agentwebmap is free to use.

### How do I add Agentwebmap to Claude Code?

Run this command in your terminal: claude mcp add --transport http agentwebmap https://agentwebmap.com/mcp

### What are the best alternatives to Agentwebmap?

The closest alternatives to Agentwebmap are Scrapling MCP Server, Firecrawl MCP Server, SearXNG Search, NeuralVerge and AI-Powered directory.

## Alternatives

- [Scrapling MCP Server](https://ai-powered.com/mcp/scrapling-mcp-server): Scrapling MCP Server is an open-source MCP server for web search and scraping.
- [Firecrawl MCP Server](https://ai-powered.com/mcp/firecrawl-mcp-server): Firecrawl MCP Server is a freemium MCP server for web search and scraping.
- [SearXNG Search](https://ai-powered.com/mcp/searxng-search): SearXNG Search is an open-source MCP server for web search and scraping.
- [NeuralVerge](https://ai-powered.com/mcp/neuralverge): NeuralVerge is an MCP server for web search and scraping.
- [AI-Powered directory](https://ai-powered.com/mcp/ai-powered-directory): AI-Powered directory is a free MCP server for developer tools.
- [BoardAgent](https://ai-powered.com/mcp/boardagent): BoardAgent is a free MCP server for community directories.
- [KitScout](https://ai-powered.com/mcp/kitscout): KitScout is a free MCP server for developer tools.
- [Shopify MCP Server (any store)](https://ai-powered.com/mcp/shopify-mcp-server-any-store): Shopify MCP Server (any store) is a paid MCP server for e-commerce.

---
Source: https://ai-powered.com/mcp/agentwebmap
