OpenMark
Benchmark your prompt on 100+ LLMs from a coding agent and pick the cheapest that passes
- Pricing
- Freemium
- Platforms
- API, Web
- Runs
- Remotely
- Version
- 1.0.0
- Listing updated
- 1 day ago
About OpenMark
OpenMark is a freemium MCP server for ai models and llm tooling. It runs as a hosted remote server, so there is nothing to install locally.
OpenMark lets a coding agent turn the prompt in your code into a repeatable test. The agent writes the task from your real prompt and data, using a YAML format with 18 scoring modes, then runs it on chosen models with real API calls through one OpenMark key. Scoring is deterministic, such as exact match, patterns, numbers and JSON checks, never an LLM judge.
Nine tools cover the loop: read the task format, validate the task for free, list current model ids and prices, upload attachments, estimate credits, run the benchmark, fetch results, export an OpenClaw routing file, and watch the task so it re-tests when new or cheaper models ship. Results report score, stability, cost per run and latency, and show which tests each model missed.
The winner can be chosen by accuracy, cost, speed or consistency, usually within a 2-point margin of the top score. Claude.ai and Claude Desktop connectors do not accept the key header yet, so use Claude Code, Cursor, Codex or VS Code.
Install
Run in a terminal
claude mcp add --transport http openmark https://openmark.ai/mcpAdd to Settings → Connectors → Add custom connector
https://openmark.ai/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"openmark": {
"url": "https://openmark.ai/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"openmark": {
"type": "http",
"url": "https://openmark.ai/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"openmark": {
"serverUrl": "https://openmark.ai/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"openmark": {
"type": "http",
"url": "https://openmark.ai/mcp"
}
}
}Tools this server provides (10)
Taken from the project documentation; the running server may differ.
- get_task_format
- Returns the task YAML format and all 18 scoring modes.
- validate_task
- Checks a task before anything runs, for free.
- list_models
- Lists current model ids, prices and image support.
- prepare_attachments
- Creates a one-hour upload link for test files.
- estimate_cost
- Estimates credits per model without running.
- run_benchmark
- Saves the task and runs it on chosen models.
- get_results
- Returns score, stability, cost and latency per model.
- export_openclaw_csv
- Exports the routing file for the OpenMark Router plugin.
- watch_task
- Re-tests the task when new or cheaper models ship.
- get_account
- Shows plan, credit balance and limits.
What it does
- Prompt-based tests
- The agent writes tests from your actual prompt and data.
- Deterministic scoring
- 18 scoring modes, with no LLM judge.
- Cost estimate first
- Quote credits per model before anything runs, with an optional budget cap.
- Priority picks
- Winner by accuracy, cost, speed or consistency.
- Watch for new models
- Re-tests when new or cheaper models appear and emails on a better winner.
- Router export
- Exports a routing CSV for the OpenMark Router plugin.
Good for
- Find the cheapest model that still extracts invoices correctly
- Compare six models on a fixed set of test cases
- Check how stable a model's answers are across runs
- Re-evaluate your prompt when a new model ships
Pricing
Freemium
50 free credits on sign-up; runs spend credits
Prices change. Check openmark.ai before you buy.
OpenMark FAQ
Do I need provider accounts?
No. One OpenMark key covers the 100+ models.
Is there a free start?
Signing up at openmark.ai gives 50 free credits; running benchmarks spends credits.
Which clients work?
Claude Code, Cursor, Codex, VS Code, OpenClaw and other clients that send a bearer header.
Does validation cost credits?
No, validate_task is free.
Is OpenMark free?
OpenMark has a free plan, with paid plans for more usage or features.
How do I add OpenMark to Claude Code?
Run this command in your terminal: claude mcp add --transport http openmark https://openmark.ai/mcp
What are the best alternatives to OpenMark?
The closest alternatives to OpenMark are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.
Alternatives to OpenMark

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Codebase Memory
MCP serverLocal MCP server that builds a code knowledge graph for AI coding agents
Runs locally- Open source
- 48.7K GitHub stars
- Knowledge and memory

GitHub
MCP serverOfficial MCP server connecting AI assistants to GitHub repos, issues and workflows
Not checked yet- Remote
- Open source
- 34.5K GitHub stars
- Developer tools

Serena MCP: the IDE for your agent
MCP serverMCP toolkit giving coding agents symbol-level code retrieval and editing
Runs locally- Open source
- 31.5K GitHub stars
- Developer tools

XERJ
MCP serverLocal search engine for AI agents with cited answers over your code and docs
Runs locally- Open source
- 19.2K GitHub stars
- Knowledge and memory

Ouroboros
MCP serverSpec-first layer that pins requirements and verifies results of AI coding agent work
Runs locally- Open source
- 6.4K GitHub stars
- Developer tools