Skip to content
MCP server

evalpair

Run LLM eval suites on local or hosted models and tell real changes from noise

Runs locallyStatus and history
Install
Pricing
Open source
Platforms
CLI, self-hosted
Runs
Locally
Version
0.1.1
Listing updated
today

Similar MCP servers

About evalpair

evalpair is an open-source MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.

Where most eval tools print two scores side by side, evalpair pairs two runs item by item, puts a bootstrap interval on the difference, counts items that got worse and better, and can measure a noise floor by asking the same model the same questions twice. The author reports that identical prompts at temperature 0 changed up to 3% of answers, so smaller changes mean little.

It talks to Ollama by default or any OpenAI-compatible server such as vLLM, llama.cpp or LM Studio, and every reply is cached so interrupted runs resume. Three suites are bundled: gsm8k-160, mmlu-hs-160 and e2e-extract-150. Custom suites use a TOML spec plus JSONL items with deterministic checkers (number, choice, exact, contains, regex, json_fields, chrF); no LLM grades another.

Install with uvx; the MCP server starts with uvx evalpair mcp over stdio. Limits: one model per run, single-turn chat prompts, no tool use. Code is MIT-licensed.

Install

Run in a terminal

claude mcp add evalpair --env EVALPAIR_API_KEY='<EVALPAIR_API_KEY>' -- uvx evalpair

Add to claude_desktop_config.json

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}

Add to .vscode/mcp.json

{
  "servers": {
    "evalpair": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}

Using several servers? Build one config file with the MCP config generator.

Configuration

  • EVALPAIR_HOMEFolder for the call cache, saved runs and your own suites (default ~/.evalpair).
  • EVALPAIR_API_KEYBearer token for an OpenAI-compatible server, if it needs one (OPENAI_API_KEY also works).

Tools this server provides (5)

Taken from the project documentation; the running server may differ.

list_suites
Lists runnable suites with name, item count, checker and what they measure.
run_eval
Runs a suite against one model and saves the run, reusing cached replies.
list_runs
Lists saved runs, newest first.
show_run
Shows a run's score with its interval, noise floor and failed replies.
compare_runs
Compares two runs paired by item, with flips each way and change versus noise floor.

What it does

Paired comparison
Differences are bootstrapped item by item, with worse and better counts.
Noise floor
--repeat measures how much answers change when the same call is repeated.
Local and hosted backends
Ollama or any OpenAI-compatible server.
Cached resumable runs
Interrupted runs continue from cached replies.
Deterministic checkers
Seven code-based checkers score answers without an LLM judge.
Custom suites
A TOML file and JSONL items define your own suite.

Good for

  • Check whether a 2-bit build really scores worse than 4-bit
  • Compare two prompts on the same 160 questions
  • Let an agent run an eval and read the verdict
  • Build a small suite for a domain-specific task

Pricing

Open source

MIT licence for the code; bundled suites keep their sources' licences.

evalpair FAQ

Which models can it call?

Ollama models or any OpenAI-compatible endpoint.

Does it need an API key?

Only if your server does; set EVALPAIR_API_KEY or OPENAI_API_KEY.

Where is data stored?

In one local folder, ~/.evalpair by default.

What are its limits?

One model per run, single-turn prompts, no tool use or constrained decoding.

Is evalpair free?

Yes. evalpair is open source, so you can use and self-host it for free.

How do I add evalpair to Claude Code?

Run this command in your terminal: claude mcp add evalpair --env EVALPAIR_API_KEY='<EVALPAIR_API_KEY>' -- uvx evalpair

What are the best alternatives to evalpair?

The closest alternatives to evalpair are headroom, omniget, next-ai-draw-io, mcp-for-blender and Claude Code Ultimate Guide.

Where this comes from

  • Description: written from the project's README.
  • Install steps: from the official MCP Registry entry.

How AIPoweredBot checks servers