Skip to content
MCP server

LLM Prover

Compare models on cost and quality, score outputs and analyse agent runs

Pricing
Paid
Platforms
Web, API
Runs
Remotely
Version
1.0.1
Listing updated
3 days ago

About LLM Prover

LLM Prover is a paid MCP server for ai models and llm tooling. It runs as a hosted remote server, so there is nothing to install locally.

LLM Prover treats model evaluation like testing: measure one model as you would a unit test, or run a whole agent workflow as an integration test. A prompt goes to several models at once and returns quality scores, cost and latency side by side, on your own data rather than a public leaderboard.

Scoring can use a gold-standard answer or custom rubric criteria, and higher plans add compliance scoring. Agent analysis reads a workflow's trace and shows where it wasted money, time or effort. Scheduled benchmarks track quality, cost and latency over time with drift alerts, and you can include private, fine-tuned or self-hosted models through any OpenAI-compatible endpoint.

The MCP server is included on the Pro plan and above. Plans are monthly with usage included and optional top-ups: Starter $49 (up to 5 models per run), Pro $129 (8), Pro Plus $199 (10) and Enterprise $499 (20). Starter has REST API access only.

Install

Run in a terminal

claude mcp add --transport http llmprover https://mcp.llmprover.pysolvr.com/mcp

Add to Settings → Connectors → Add custom connector

https://mcp.llmprover.pysolvr.com/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "llmprover": {
      "url": "https://mcp.llmprover.pysolvr.com/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "llmprover": {
      "type": "http",
      "url": "https://mcp.llmprover.pysolvr.com/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "llmprover": {
      "serverUrl": "https://mcp.llmprover.pysolvr.com/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "llmprover": {
      "type": "http",
      "url": "https://mcp.llmprover.pysolvr.com/mcp"
    }
  }
}

What it does

Side-by-side comparison
Run one prompt across models and see quality, cost and latency together.
Gold standard and rubrics
Score each response against a reference answer or custom criteria.
Agent analysis
Find where an agent workflow wasted money, time or effort.
Scheduled benchmarks
Track drift over time with alerts, from daily to hourly by plan.
Bring your own endpoint
Benchmark any OpenAI-compatible model alongside hosted ones.

Good for

  • Pick the cheapest model that meets a quality bar
  • Score outputs against a rubric without manual review
  • Find wasteful steps in an agent run
  • Watch a production prompt for quality drift

Pricing

Paid, from $49 a month

Starter $49/mo, Pro $129/mo, Pro Plus $199/mo, Enterprise $499/mo; MCP from Pro

Prices change. Check llmprover.pysolvr.com before you buy.

LLM Prover FAQ

Which plan includes the MCP server?

Pro and above; Starter has REST API access only.

How many models can I compare?

Up to 5 on Starter, 8 on Pro, 10 on Pro Plus and 20 on Enterprise.

Can I test my own models?

Yes, through any OpenAI-compatible endpoint.

What does it cost?

Plans run $49, $129, $199 and $499 a month, with usage included and top-ups available.

Is LLM Prover free?

No. LLM Prover is a paid product. Paid plans start at $49 a month.

How do I add LLM Prover to Claude Code?

Run this command in your terminal: claude mcp add --transport http llmprover https://mcp.llmprover.pysolvr.com/mcp

What are the best alternatives to LLM Prover?

The closest alternatives to LLM Prover are Claude Code Ultimate Guide, signafAIde, Toolkien, Lugh and HypePaper.

Alternatives to LLM Prover

  1. MCP server for searching the Claude Code Ultimate Guide from any MCP client

    Runs locally
    • Open source
    • 6.5K GitHub stars
    • AI models and LLM tooling
  2. signafAIde

    MCP server

    Local MCP server to set up, verify and check coverage of signed agent action records

    Runs locally
    • 5 tools
    • Freemium
    • Security
  3. Toolkien

    MCP server

    One MCP gateway for all your MCP servers, with code mode, permissions and audit

    Not checked yet
    • 3 tools
    • Remote
    • Open source
    • AI models and LLM tooling
  4. Lugh

    MCP server

    Write, check and dry-run Lugh agent programs with policy, budgets and approvals

    Runs locally
    • 5 tools
    • Open source
    • AI models and LLM tooling
  5. HypePaper

    MCP server

    Research papers, benchmarks, citations and topic experts for AI assistants

    • 58 tools
    • Remote
    • Freemium
    • AI models and LLM tooling