evalpair
Run LLM eval suites on local or hosted models and tell real changes from noise
- Pricing
- Open source
- Platforms
- CLI, self-hosted
- Runs
- Locally
- Version
- 0.1.1
- Listing updated
- today
Similar MCP servers
About evalpair
evalpair is an open-source MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.
Where most eval tools print two scores side by side, evalpair pairs two runs item by item, puts a bootstrap interval on the difference, counts items that got worse and better, and can measure a noise floor by asking the same model the same questions twice. The author reports that identical prompts at temperature 0 changed up to 3% of answers, so smaller changes mean little.
It talks to Ollama by default or any OpenAI-compatible server such as vLLM, llama.cpp or LM Studio, and every reply is cached so interrupted runs resume. Three suites are bundled: gsm8k-160, mmlu-hs-160 and e2e-extract-150. Custom suites use a TOML spec plus JSONL items with deterministic checkers (number, choice, exact, contains, regex, json_fields, chrF); no LLM grades another.
Install with uvx; the MCP server starts with uvx evalpair mcp over stdio. Limits: one model per run, single-turn chat prompts, no tool use. Code is MIT-licensed.
Install
Run in a terminal
claude mcp add evalpair --env EVALPAIR_API_KEY='<EVALPAIR_API_KEY>' -- uvx evalpairAdd to claude_desktop_config.json
{
"mcpServers": {
"evalpair": {
"command": "uvx",
"args": [
"evalpair"
],
"env": {
"EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
}
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"evalpair": {
"command": "uvx",
"args": [
"evalpair"
],
"env": {
"EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
}
}
}
}Add to .vscode/mcp.json
{
"servers": {
"evalpair": {
"type": "stdio",
"command": "uvx",
"args": [
"evalpair"
],
"env": {
"EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
}
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"evalpair": {
"command": "uvx",
"args": [
"evalpair"
],
"env": {
"EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
}
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"evalpair": {
"command": "uvx",
"args": [
"evalpair"
],
"env": {
"EVALPAIR_API_KEY": "<EVALPAIR_API_KEY>"
}
}
}
}Using several servers? Build one config file with the MCP config generator.
Configuration
EVALPAIR_HOMEFolder for the call cache, saved runs and your own suites (default ~/.evalpair).EVALPAIR_API_KEYBearer token for an OpenAI-compatible server, if it needs one (OPENAI_API_KEY also works).
Tools this server provides (5)
Taken from the project documentation; the running server may differ.
- list_suites
- Lists runnable suites with name, item count, checker and what they measure.
- run_eval
- Runs a suite against one model and saves the run, reusing cached replies.
- list_runs
- Lists saved runs, newest first.
- show_run
- Shows a run's score with its interval, noise floor and failed replies.
- compare_runs
- Compares two runs paired by item, with flips each way and change versus noise floor.
What it does
- Paired comparison
- Differences are bootstrapped item by item, with worse and better counts.
- Noise floor
- --repeat measures how much answers change when the same call is repeated.
- Local and hosted backends
- Ollama or any OpenAI-compatible server.
- Cached resumable runs
- Interrupted runs continue from cached replies.
- Deterministic checkers
- Seven code-based checkers score answers without an LLM judge.
- Custom suites
- A TOML file and JSONL items define your own suite.
Good for
- Check whether a 2-bit build really scores worse than 4-bit
- Compare two prompts on the same 160 questions
- Let an agent run an eval and read the verdict
- Build a small suite for a domain-specific task
Pricing
Open source
MIT licence for the code; bundled suites keep their sources' licences.
evalpair FAQ
Which models can it call?
Ollama models or any OpenAI-compatible endpoint.
Does it need an API key?
Only if your server does; set EVALPAIR_API_KEY or OPENAI_API_KEY.
Where is data stored?
In one local folder, ~/.evalpair by default.
What are its limits?
One model per run, single-turn prompts, no tool use or constrained decoding.
Is evalpair free?
Yes. evalpair is open source, so you can use and self-host it for free.
How do I add evalpair to Claude Code?
Run this command in your terminal: claude mcp add evalpair --env EVALPAIR_API_KEY='<EVALPAIR_API_KEY>' -- uvx evalpair
What are the best alternatives to evalpair?
The closest alternatives to evalpair are headroom, omniget, next-ai-draw-io, mcp-for-blender and Claude Code Ultimate Guide.
Where this comes from
- Description: written from the project's README.
- Install steps: from the official MCP Registry entry.



