
LLM Prover
Compare models on cost and quality, score outputs and analyse agent runs
- Pricing
- Paid
- Platforms
- Web, API
- Runs
- Remotely
- Version
- 1.0.1
- Listing updated
- 3 days ago
About LLM Prover
LLM Prover is a paid MCP server for ai models and llm tooling. It runs as a hosted remote server, so there is nothing to install locally.
LLM Prover treats model evaluation like testing: measure one model as you would a unit test, or run a whole agent workflow as an integration test. A prompt goes to several models at once and returns quality scores, cost and latency side by side, on your own data rather than a public leaderboard.
Scoring can use a gold-standard answer or custom rubric criteria, and higher plans add compliance scoring. Agent analysis reads a workflow's trace and shows where it wasted money, time or effort. Scheduled benchmarks track quality, cost and latency over time with drift alerts, and you can include private, fine-tuned or self-hosted models through any OpenAI-compatible endpoint.
The MCP server is included on the Pro plan and above. Plans are monthly with usage included and optional top-ups: Starter $49 (up to 5 models per run), Pro $129 (8), Pro Plus $199 (10) and Enterprise $499 (20). Starter has REST API access only.
Install
Run in a terminal
claude mcp add --transport http llmprover https://mcp.llmprover.pysolvr.com/mcpAdd to Settings → Connectors → Add custom connector
https://mcp.llmprover.pysolvr.com/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"llmprover": {
"url": "https://mcp.llmprover.pysolvr.com/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"llmprover": {
"type": "http",
"url": "https://mcp.llmprover.pysolvr.com/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"llmprover": {
"serverUrl": "https://mcp.llmprover.pysolvr.com/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"llmprover": {
"type": "http",
"url": "https://mcp.llmprover.pysolvr.com/mcp"
}
}
}What it does
- Side-by-side comparison
- Run one prompt across models and see quality, cost and latency together.
- Gold standard and rubrics
- Score each response against a reference answer or custom criteria.
- Agent analysis
- Find where an agent workflow wasted money, time or effort.
- Scheduled benchmarks
- Track drift over time with alerts, from daily to hourly by plan.
- Bring your own endpoint
- Benchmark any OpenAI-compatible model alongside hosted ones.
Good for
- Pick the cheapest model that meets a quality bar
- Score outputs against a rubric without manual review
- Find wasteful steps in an agent run
- Watch a production prompt for quality drift
Pricing
Paid, from $49 a month
Starter $49/mo, Pro $129/mo, Pro Plus $199/mo, Enterprise $499/mo; MCP from Pro
Prices change. Check llmprover.pysolvr.com before you buy.
LLM Prover FAQ
Which plan includes the MCP server?
Pro and above; Starter has REST API access only.
How many models can I compare?
Up to 5 on Starter, 8 on Pro, 10 on Pro Plus and 20 on Enterprise.
Can I test my own models?
Yes, through any OpenAI-compatible endpoint.
What does it cost?
Plans run $49, $129, $199 and $499 a month, with usage included and top-ups available.
Is LLM Prover free?
No. LLM Prover is a paid product. Paid plans start at $49 a month.
How do I add LLM Prover to Claude Code?
Run this command in your terminal: claude mcp add --transport http llmprover https://mcp.llmprover.pysolvr.com/mcp
What are the best alternatives to LLM Prover?
The closest alternatives to LLM Prover are Claude Code Ultimate Guide, signafAIde, Toolkien, Lugh and HypePaper.
Alternatives to LLM Prover

Claude Code Ultimate Guide
MCP serverMCP server for searching the Claude Code Ultimate Guide from any MCP client
Runs locally- Open source
- 6.5K GitHub stars
- AI models and LLM tooling

signafAIde
MCP serverLocal MCP server to set up, verify and check coverage of signed agent action records
Runs locally- 5 tools
- Freemium
- Security

Toolkien
MCP serverOne MCP gateway for all your MCP servers, with code mode, permissions and audit
Not checked yet- 3 tools
- Remote
- Open source
- AI models and LLM tooling
Lugh
MCP serverWrite, check and dry-run Lugh agent programs with policy, budgets and approvals
Runs locally- 5 tools
- Open source
- AI models and LLM tooling
HypePaper
MCP serverResearch papers, benchmarks, citations and topic experts for AI assistants
- 58 tools
- Remote
- Freemium
- AI models and LLM tooling

One compact entry point that finds, describes and runs hundreds of pay-per-call data tools
- 3 tools
- Remote
- Freemium
- Developer tools