Skip to content
MCP server

Nerf Watch

Daily scores for Claude, GPT and other models on one private, code-graded test suite

Install
Pricing
Free
Platforms
API, Web
Runs
Remotely
Version
1.0.0
Listing updated
2 days ago

About Nerf Watch

Nerf Watch is a free MCP server for ai models and llm tooling. It runs as a hosted remote server, so there is nothing to install locally.

Nerf Watch tracks whether coding models get worse over time. Each day the same private tests run against Claude models through Claude Code, OpenAI models through Codex, and several others through the OpenCode CLI, with code doing the grading. The questions stay private to limit training on them, and a marker appears on a model's chart when a score jumps more than 10 points between runs.

The MCP server exposes the same public data read-only. One tool returns latest scores with previous runs, changes, 7-day baselines and trends, including model ids. The other takes a model id and returns its retained history, oldest first, with the suite version for each point. Missing days are missing rather than zero, and comparisons are made within one suite version. Test questions and grader details are not returned.

The README cautions that a lower score is a lower score on this suite, not proof that a provider changed a model, since runs are noisy. Both tools share a 300-second cache, and shown data needs a Data: tokenusage.dev credit. The service is unofficial.

Install

Run in a terminal

claude mcp add --transport http nerfwatch https://tokenusage.dev/mcp/nerfwatch

Add to Settings → Connectors → Add custom connector

https://tokenusage.dev/mcp/nerfwatch
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "nerfwatch": {
      "url": "https://tokenusage.dev/mcp/nerfwatch"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "nerfwatch": {
      "type": "http",
      "url": "https://tokenusage.dev/mcp/nerfwatch"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "nerfwatch": {
      "serverUrl": "https://tokenusage.dev/mcp/nerfwatch"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "nerfwatch": {
      "type": "http",
      "url": "https://tokenusage.dev/mcp/nerfwatch"
    }
  }
}

Tools this server provides (2)

Taken from the project documentation; the running server may differ.

get_nerfwatch_scores
Returns the latest public scores, previous runs, changes, baselines and trends with model ids.
get_nerfwatch_history
Returns one model's retained score history, oldest first, with suite versions.

What it does

Latest scores
Returns each model's current score, change from the last run and 7-day baseline.
Model history
Returns one model's retained scores, oldest first, with suite version.
Trend labels
Scores within 2 points of the baseline read as steady.
Private suite
Test questions stay hidden to reduce training on them.
No key needed
Hosted, read-only, no account.
Speed probe
The site also shows daily response times by model.

Good for

  • Ask whether a model's quality dipped this week
  • Compare Claude and GPT models on the same suite
  • Chart a model's score history in a dashboard
  • Check if a score change is within the usual noise

Pricing

Free

Free, read-only; credit link required for public display

Prices change. Check tokenusage.dev before you buy.

Nerf Watch FAQ

Does a lower score mean a model was changed?

No. The README says it is only a lower score on this suite, as runs are noisy.

What do I need to use it?

No key or account.

How often is it updated?

Tests run once a day, and responses are cached for 300 seconds.

Can I show the data publicly?

Yes, with a visible Data: tokenusage.dev credit linking to the Nerf Watch page.

Is Nerf Watch free?

Yes. Nerf Watch is free to use.

How do I add Nerf Watch to Claude Code?

Run this command in your terminal: claude mcp add --transport http nerfwatch https://tokenusage.dev/mcp/nerfwatch

What are the best alternatives to Nerf Watch?

The closest alternatives to Nerf Watch are Claude Code Ultimate Guide, signafAIde, Toolkien, Lugh and HypePaper.

Alternatives to Nerf Watch

  1. MCP server for searching the Claude Code Ultimate Guide from any MCP client

    Runs locally
    • Open source
    • 6.5K GitHub stars
    • AI models and LLM tooling
  2. signafAIde

    MCP server

    Local MCP server to set up, verify and check coverage of signed agent action records

    Runs locally
    • 5 tools
    • Freemium
    • Security
  3. Toolkien

    MCP server

    One MCP gateway for all your MCP servers, with code mode, permissions and audit

    Not checked yet
    • 3 tools
    • Remote
    • Open source
    • AI models and LLM tooling
  4. Lugh

    MCP server

    Write, check and dry-run Lugh agent programs with policy, budgets and approvals

    Runs locally
    • 5 tools
    • Open source
    • AI models and LLM tooling
  5. HypePaper

    MCP server

    Research papers, benchmarks, citations and topic experts for AI assistants

    • 58 tools
    • Remote
    • Freemium
    • AI models and LLM tooling