Skip to content
MCP server

OpenMark

Benchmark your prompt on 100+ LLMs from a coding agent and pick the cheapest that passes

Install
Pricing
Freemium
Platforms
API, Web
Runs
Remotely
Version
1.0.0
Listing updated
1 day ago

About OpenMark

OpenMark is a freemium MCP server for ai models and llm tooling. It runs as a hosted remote server, so there is nothing to install locally.

OpenMark lets a coding agent turn the prompt in your code into a repeatable test. The agent writes the task from your real prompt and data, using a YAML format with 18 scoring modes, then runs it on chosen models with real API calls through one OpenMark key. Scoring is deterministic, such as exact match, patterns, numbers and JSON checks, never an LLM judge.

Nine tools cover the loop: read the task format, validate the task for free, list current model ids and prices, upload attachments, estimate credits, run the benchmark, fetch results, export an OpenClaw routing file, and watch the task so it re-tests when new or cheaper models ship. Results report score, stability, cost per run and latency, and show which tests each model missed.

The winner can be chosen by accuracy, cost, speed or consistency, usually within a 2-point margin of the top score. Claude.ai and Claude Desktop connectors do not accept the key header yet, so use Claude Code, Cursor, Codex or VS Code.

Install

Run in a terminal

claude mcp add --transport http openmark https://openmark.ai/mcp

Add to Settings → Connectors → Add custom connector

https://openmark.ai/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "openmark": {
      "url": "https://openmark.ai/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "openmark": {
      "type": "http",
      "url": "https://openmark.ai/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "openmark": {
      "serverUrl": "https://openmark.ai/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "openmark": {
      "type": "http",
      "url": "https://openmark.ai/mcp"
    }
  }
}

Tools this server provides (10)

Taken from the project documentation; the running server may differ.

get_task_format
Returns the task YAML format and all 18 scoring modes.
validate_task
Checks a task before anything runs, for free.
list_models
Lists current model ids, prices and image support.
prepare_attachments
Creates a one-hour upload link for test files.
estimate_cost
Estimates credits per model without running.
run_benchmark
Saves the task and runs it on chosen models.
get_results
Returns score, stability, cost and latency per model.
export_openclaw_csv
Exports the routing file for the OpenMark Router plugin.
watch_task
Re-tests the task when new or cheaper models ship.
get_account
Shows plan, credit balance and limits.

What it does

Prompt-based tests
The agent writes tests from your actual prompt and data.
Deterministic scoring
18 scoring modes, with no LLM judge.
Cost estimate first
Quote credits per model before anything runs, with an optional budget cap.
Priority picks
Winner by accuracy, cost, speed or consistency.
Watch for new models
Re-tests when new or cheaper models appear and emails on a better winner.
Router export
Exports a routing CSV for the OpenMark Router plugin.

Good for

  • Find the cheapest model that still extracts invoices correctly
  • Compare six models on a fixed set of test cases
  • Check how stable a model's answers are across runs
  • Re-evaluate your prompt when a new model ships

Pricing

Freemium

50 free credits on sign-up; runs spend credits

Prices change. Check openmark.ai before you buy.

OpenMark FAQ

Do I need provider accounts?

No. One OpenMark key covers the 100+ models.

Is there a free start?

Signing up at openmark.ai gives 50 free credits; running benchmarks spends credits.

Which clients work?

Claude Code, Cursor, Codex, VS Code, OpenClaw and other clients that send a bearer header.

Does validation cost credits?

No, validate_task is free.

Is OpenMark free?

OpenMark has a free plan, with paid plans for more usage or features.

How do I add OpenMark to Claude Code?

Run this command in your terminal: claude mcp add --transport http openmark https://openmark.ai/mcp

What are the best alternatives to OpenMark?

The closest alternatives to OpenMark are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.

Alternatives to OpenMark

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  3. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  4. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  5. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory
  6. Ouroboros

    MCP server

    Spec-first layer that pins requirements and verifies results of AI coding agent work

    Runs locally
    • Open source
    • 6.4K GitHub stars
    • Developer tools