Skip to content
MCP server

Modelwatch

Create, run and review LLM behavioral drift specs from your AI client

Runs locallyStatus and history
Pricing
Freemium
Platforms
Web, API, CLI
Runs
Locally
Version
0.1.3
Listing updated
1 day ago

About Modelwatch

Modelwatch is a freemium MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.

ModelWatch exists to catch silent provider-side model updates that change an app's output between your own test runs. A spec holds an input prompt, a schedule and expectation rules; the first run sets the baseline, and later hourly, daily or weekly runs are scored against it. The site describes a five-axis diff weighting semantic similarity, format, refusal, length and required text, with alerts bucketed from low to critical.

This MCP server lets a client manage that without leaving chat. Nine tools list and create endpoints and specs, run a spec on demand, reset a baseline after an intentional model swap, read recent drift events, view a spec's history and check health. The README's example creates a refusal canary against an OpenAI endpoint and reports the baseline.

Users bring their own OpenAI or Anthropic key, so ModelWatch adds no token markup. The free tier includes 5 specs and 500 runs a month; Pro is $99 a month and Team $299. The server runs over stdio with MODELWATCH_API_KEY, and a self-hosted API base is accepted over HTTPS.

Install

Run in a terminal

claude mcp add modelwatch --env MODELWATCH_API_KEY='<MODELWATCH_API_KEY>' -- npx -y modelwatch-mcp

Add to claude_desktop_config.json

{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}

Add to .vscode/mcp.json

{
  "servers": {
    "modelwatch": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ],
      "env": {
        "MODELWATCH_API_KEY": "<MODELWATCH_API_KEY>"
      }
    }
  }
}

Configuration

  • MODELWATCH_API_KEYAPI key from https://modelwatch.app (free tier: 5 specs, 500 runs/mo).Required
  • MODELWATCH_API_BASEOverride the API base URL for self-hosted deployments.

Tools this server provides (9)

Taken from the project documentation; the running server may differ.

list_endpoints
Shows monitored LLM endpoints.
create_endpoint
Adds an endpoint with provider and model.
list_specs
Shows behavioral specs.
create_spec
Adds a spec with prompt, schedule and optional similarity threshold.
run_spec
Runs a spec on demand, setting the baseline on first run.
reset_baseline
Re-baselines after an intentional model swap.
get_drift_events
Shows recent drift events.
get_spec_history
Shows run history for one spec.
get_health
Shows per-spec health, last drift score and last run time.

What it does

Scheduled drift specs
Replay prompts hourly, daily or weekly against a baseline.
Five-axis scoring
Weights semantic, format, refusal, length and contains checks.
Baseline reset
Re-baseline after an intentional model swap.
Drift events and history
Read recent events and per-spec run history.
Bring your own key
Uses your provider key with no token markup.
Tool monitoring
A decorator tracks latency and response structure of tools.

Good for

  • Set up a refusal canary against an LLM endpoint
  • Find out when a provider quietly changes a model
  • Re-baseline after switching model versions
  • Check which specs are healthy from chat

Pricing

Freemium, from $99 a month

Free tier; Pro $99/mo, Team $299/mo

Prices change. Check modelwatch.app before you buy.

Modelwatch FAQ

Is there a free tier?

Yes: 5 specs, 500 runs a month and 5,000 tool calls, with email alerts.

Do they charge for LLM tokens?

No. You supply your own OpenAI or Anthropic key.

How do I install the MCP server?

npm install -g modelwatch-mcp and set MODELWATCH_API_KEY.

What do paid plans cost?

Pro is $99 a month, Team $299 and Enterprise $999.

Is Modelwatch free?

Modelwatch has a free plan, with paid plans for more usage or features. Paid plans start at $99 a month.

How do I add Modelwatch to Claude Code?

Run this command in your terminal: claude mcp add modelwatch --env MODELWATCH_API_KEY='<MODELWATCH_API_KEY>' -- npx -y modelwatch-mcp

What are the best alternatives to Modelwatch?

The closest alternatives to Modelwatch are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.

Alternatives to Modelwatch

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  3. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  4. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  5. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory
  6. Ouroboros

    MCP server

    Spec-first layer that pins requirements and verifies results of AI coding agent work

    Runs locally
    • Open source
    • 6.4K GitHub stars
    • Developer tools