Modelwatch
Create, run and review LLM behavioral drift specs from your AI client
- Pricing
- Freemium
- Platforms
- Web, API, CLI
- Runs
- Locally
- Version
- 0.1.3
- Listing updated
- 1 day ago
About Modelwatch
Modelwatch is a freemium MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.
ModelWatch exists to catch silent provider-side model updates that change an app's output between your own test runs. A spec holds an input prompt, a schedule and expectation rules; the first run sets the baseline, and later hourly, daily or weekly runs are scored against it. The site describes a five-axis diff weighting semantic similarity, format, refusal, length and required text, with alerts bucketed from low to critical.
This MCP server lets a client manage that without leaving chat. Nine tools list and create endpoints and specs, run a spec on demand, reset a baseline after an intentional model swap, read recent drift events, view a spec's history and check health. The README's example creates a refusal canary against an OpenAI endpoint and reports the baseline.
Users bring their own OpenAI or Anthropic key, so ModelWatch adds no token markup. The free tier includes 5 specs and 500 runs a month; Pro is $99 a month and Team $299. The server runs over stdio with MODELWATCH_API_KEY, and a self-hosted API base is accepted over HTTPS.
Install
Run in a terminal
claude mcp add modelwatch --env MODELWATCH_API_KEY='<MODELWATCH_API_KEY>' -- npx -y modelwatch-mcpAdd to claude_desktop_config.json
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
}
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
}
}
}
}Add to .vscode/mcp.json
{
"servers": {
"modelwatch": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
}
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
}
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
}
}
}
}Configuration
MODELWATCH_API_KEYAPI key from https://modelwatch.app (free tier: 5 specs, 500 runs/mo).RequiredMODELWATCH_API_BASEOverride the API base URL for self-hosted deployments.
Tools this server provides (9)
Taken from the project documentation; the running server may differ.
- list_endpoints
- Shows monitored LLM endpoints.
- create_endpoint
- Adds an endpoint with provider and model.
- list_specs
- Shows behavioral specs.
- create_spec
- Adds a spec with prompt, schedule and optional similarity threshold.
- run_spec
- Runs a spec on demand, setting the baseline on first run.
- reset_baseline
- Re-baselines after an intentional model swap.
- get_drift_events
- Shows recent drift events.
- get_spec_history
- Shows run history for one spec.
- get_health
- Shows per-spec health, last drift score and last run time.
What it does
- Scheduled drift specs
- Replay prompts hourly, daily or weekly against a baseline.
- Five-axis scoring
- Weights semantic, format, refusal, length and contains checks.
- Baseline reset
- Re-baseline after an intentional model swap.
- Drift events and history
- Read recent events and per-spec run history.
- Bring your own key
- Uses your provider key with no token markup.
- Tool monitoring
- A decorator tracks latency and response structure of tools.
Good for
- Set up a refusal canary against an LLM endpoint
- Find out when a provider quietly changes a model
- Re-baseline after switching model versions
- Check which specs are healthy from chat
Pricing
Freemium, from $99 a month
Free tier; Pro $99/mo, Team $299/mo
Prices change. Check modelwatch.app before you buy.
Modelwatch FAQ
Is there a free tier?
Yes: 5 specs, 500 runs a month and 5,000 tool calls, with email alerts.
Do they charge for LLM tokens?
No. You supply your own OpenAI or Anthropic key.
How do I install the MCP server?
npm install -g modelwatch-mcp and set MODELWATCH_API_KEY.
What do paid plans cost?
Pro is $99 a month, Team $299 and Enterprise $999.
Is Modelwatch free?
Modelwatch has a free plan, with paid plans for more usage or features. Paid plans start at $99 a month.
How do I add Modelwatch to Claude Code?
Run this command in your terminal: claude mcp add modelwatch --env MODELWATCH_API_KEY='<MODELWATCH_API_KEY>' -- npx -y modelwatch-mcp
What are the best alternatives to Modelwatch?
The closest alternatives to Modelwatch are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.
Alternatives to Modelwatch

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Codebase Memory
MCP serverLocal MCP server that builds a code knowledge graph for AI coding agents
Runs locally- Open source
- 48.7K GitHub stars
- Knowledge and memory

GitHub
MCP serverOfficial MCP server connecting AI assistants to GitHub repos, issues and workflows
Not checked yet- Remote
- Open source
- 34.5K GitHub stars
- Developer tools

Serena MCP: the IDE for your agent
MCP serverMCP toolkit giving coding agents symbol-level code retrieval and editing
Runs locally- Open source
- 31.5K GitHub stars
- Developer tools

XERJ
MCP serverLocal search engine for AI agents with cited answers over your code and docs
Runs locally- Open source
- 19.2K GitHub stars
- Knowledge and memory

Ouroboros
MCP serverSpec-first layer that pins requirements and verifies results of AI coding agent work
Runs locally- Open source
- 6.4K GitHub stars
- Developer tools