# Modelwatch

> Modelwatch is a freemium MCP server for ai models and llm tooling. Create, run and review LLM behavioral drift specs from your AI client. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Freemium
- **Starting price:** $99 a month
- **Platforms:** Web, API, CLI, Slack
- **Website:** https://modelwatch.app
- **Repository:** https://github.com/bch1212/modelwatch/tree/HEAD/mcp-server
- **MCP Registry name:** io.github.bch1212/modelwatch
- **Categories:** AI models and LLM tooling, developer tools
- **Listing updated:** 2026-11-16

## About

ModelWatch exists to catch silent provider-side model updates that change an app's output between your own test runs. A spec holds an input prompt, a schedule and expectation rules; the first run sets the baseline, and later hourly, daily or weekly runs are scored against it. The site describes a five-axis diff weighting semantic similarity, format, refusal, length and required text, with alerts bucketed from low to critical.

This MCP server lets a client manage that without leaving chat. Nine tools list and create endpoints and specs, run a spec on demand, reset a baseline after an intentional model swap, read recent drift events, view a spec's history and check health. The README's example creates a refusal canary against an OpenAI endpoint and reports the baseline.

Users bring their own OpenAI or Anthropic key, so ModelWatch adds no token markup. The free tier includes 5 specs and 500 runs a month; Pro is $99 a month and Team $299. The server runs over stdio with MODELWATCH_API_KEY, and a self-hosted API base is accepted over HTTPS.

## Features

- **Scheduled drift specs:** Replay prompts hourly, daily or weekly against a baseline.
- **Five-axis scoring:** Weights semantic, format, refusal, length and contains checks.
- **Baseline reset:** Re-baseline after an intentional model swap.
- **Drift events and history:** Read recent events and per-spec run history.
- **Bring your own key:** Uses your provider key with no token markup.
- **Tool monitoring:** A decorator tracks latency and response structure of tools.

## Good for

- Set up a refusal canary against an LLM endpoint
- Find out when a provider quietly changes a model
- Re-baseline after switching model versions
- Check which specs are healthy from chat

## Install

### Claude Code (Terminal)

```sh
claude mcp add modelwatch --env MODELWATCH_API_KEY='<MODELWATCH_API_KEY>' -- npx -y modelwatch-mcp
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "modelwatch": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}
```

## Tools

- `list_endpoints`: Shows monitored LLM endpoints.
- `create_endpoint`: Adds an endpoint with provider and model.
- `list_specs`: Shows behavioral specs.
- `create_spec`: Adds a spec with prompt, schedule and optional similarity threshold.
- `run_spec`: Runs a spec on demand, setting the baseline on first run.
- `reset_baseline`: Re-baselines after an intentional model swap.
- `get_drift_events`: Shows recent drift events.
- `get_spec_history`: Shows run history for one spec.
- `get_health`: Shows per-spec health, last drift score and last run time.

## FAQ

### Is there a free tier?

Yes: 5 specs, 500 runs a month and 5,000 tool calls, with email alerts.

### Do they charge for LLM tokens?

No. You supply your own OpenAI or Anthropic key.

### How do I install the MCP server?

npm install -g modelwatch-mcp and set MODELWATCH_API_KEY.

### What do paid plans cost?

Pro is $99 a month, Team $299 and Enterprise $999.

### Is Modelwatch free?

Modelwatch has a free plan, with paid plans for more usage or features. Paid plans start at $99 a month.

### How do I add Modelwatch to Claude Code?

Run this command in your terminal: claude mcp add modelwatch --env MODELWATCH_API_KEY='<MODELWATCH_API_KEY>' -- npx -y modelwatch-mcp

### What are the best alternatives to Modelwatch?

The closest alternatives to Modelwatch are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.

## Alternatives

- [REA](https://ai-powered.com/mcp/rea): REA is an open-source MCP server for developer tools.
- [Codebase Memory](https://ai-powered.com/mcp/codebase-memory): Codebase Memory is an open-source MCP server for knowledge and memory.
- [GitHub](https://ai-powered.com/mcp/github-2): GitHub is an open-source MCP server for developer tools.
- [Serena MCP: the IDE for your agent](https://ai-powered.com/mcp/serena-mcp-the-ide-for-your-agent): Serena MCP: the IDE for your agent is an open-source MCP server for developer tools.
- [XERJ](https://ai-powered.com/mcp/xerj): XERJ is an open-source MCP server for knowledge and memory.
- [Ouroboros](https://ai-powered.com/mcp/ouroboros): Ouroboros is an open-source MCP server for developer tools.
- [Claude Code Ultimate Guide](https://ai-powered.com/mcp/claude-code-ultimate-guide): Claude Code Ultimate Guide is an open-source MCP server for ai models and llm tooling.
- [Concord MCP](https://ai-powered.com/mcp/concord-mcp): Concord MCP is a freemium MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/modelwatch
