# evalpair

> evalpair is an open-source MCP server for ai models and llm tooling. Run LLM eval suites on local or hosted models and tell real changes from noise. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI, self-hosted
- **Repository:** https://github.com/srikarmk/evalpair
- **MCP Registry name:** io.github.Srikarmk/evalpair
- **Categories:** [AI models and LLM tooling](https://ai-powered.com/categories/ai-and-llm)
- **Listing updated:** 2026-11-18

## About

Where most eval tools print two scores side by side, evalpair pairs two runs item by item, puts a bootstrap interval on the difference, counts items that got worse and better, and can measure a noise floor by asking the same model the same questions twice. The author reports that identical prompts at temperature 0 changed up to 3% of answers, so smaller changes mean little.

It talks to Ollama by default or any OpenAI-compatible server such as vLLM, llama.cpp or LM Studio, and every reply is cached so interrupted runs resume. Three suites are bundled: gsm8k-160, mmlu-hs-160 and e2e-extract-150. Custom suites use a TOML spec plus JSONL items with deterministic checkers (number, choice, exact, contains, regex, json_fields, chrF); no LLM grades another.

Install with uvx; the MCP server starts with uvx evalpair mcp over stdio. Limits: one model per run, single-turn chat prompts, no tool use. Code is MIT-licensed.

## Features

- **Paired comparison:** Differences are bootstrapped item by item, with worse and better counts.
- **Noise floor:** --repeat measures how much answers change when the same call is repeated.
- **Local and hosted backends:** Ollama or any OpenAI-compatible server.
- **Cached resumable runs:** Interrupted runs continue from cached replies.
- **Deterministic checkers:** Seven code-based checkers score answers without an LLM judge.
- **Custom suites:** A TOML file and JSONL items define your own suite.

## Good for

- Check whether a 2-bit build really scores worse than 4-bit
- Compare two prompts on the same 160 questions
- Let an agent run an eval and read the verdict
- Build a small suite for a domain-specific task

## Install

### Claude Code (Terminal)

```sh
claude mcp add evalpair --env EVALPAIR_API_KEY='<EVALPAIR_API_KEY>' -- uvx evalpair
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "evalpair": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
      }
    }
  }
}
```

## Tools

- `list_suites`: Lists runnable suites with name, item count, checker and what they measure.
- `run_eval`: Runs a suite against one model and saves the run, reusing cached replies.
- `list_runs`: Lists saved runs, newest first.
- `show_run`: Shows a run's score with its interval, noise floor and failed replies.
- `compare_runs`: Compares two runs paired by item, with flips each way and change versus noise floor.

## FAQ

### Which models can it call?

Ollama models or any OpenAI-compatible endpoint.

### Does it need an API key?

Only if your server does; set EVALPAIR_API_KEY or OPENAI_API_KEY.

### Where is data stored?

In one local folder, ~/.evalpair by default.

### What are its limits?

One model per run, single-turn prompts, no tool use or constrained decoding.

### Is evalpair free?

Yes. evalpair is open source, so you can use and self-host it for free.

### How do I add evalpair to Claude Code?

Run this command in your terminal: claude mcp add evalpair --env EVALPAIR_API_KEY='<EVALPAIR_API_KEY>' -- uvx evalpair

### What are the best alternatives to evalpair?

The closest alternatives to evalpair are headroom, omniget, next-ai-draw-io, mcp-for-blender and Claude Code Ultimate Guide.

## Where this comes from

- Description: written from the project's README.
- Install steps: from the official MCP Registry entry.

## Alternatives

- [headroom](https://ai-powered.com/mcp/headroom-3): headroom is an open-source MCP server for ai models and llm tooling.
- [omniget](https://ai-powered.com/mcp/omniget): omniget is an open-source MCP server for ai models and llm tooling.
- [next-ai-draw-io](https://ai-powered.com/mcp/next-ai-draw-io-2): next-ai-draw-io is an open-source MCP server for ai models and llm tooling.
- [mcp-for-blender](https://ai-powered.com/mcp/mcp-for-blender): mcp-for-blender is an open-source MCP server for design and media.
- [Claude Code Ultimate Guide](https://ai-powered.com/mcp/claude-code-ultimate-guide): Claude Code Ultimate Guide is an open-source MCP server for ai models and llm tooling.
- [unity-mcp](https://ai-powered.com/mcp/unity-mcp-2): unity-mcp is an open-source MCP server for ai models and llm tooling.
- [fastapi_mcp](https://ai-powered.com/mcp/fastapi-mcp): fastapi_mcp is an open-source MCP server for ai models and llm tooling.
- [tyk](https://ai-powered.com/mcp/tyk): tyk is an open-source MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/evalpair
