Nonobench
Remote MCP endpoint for the Nonobench LLM nonogram puzzle benchmark
- Pricing
- Open source
- Platforms
- Web, API
- Runs
- Remotely
- Version
- 1.1.0
- Listing updated
- 1 day ago
About Nonobench
Nonobench is an open-source MCP server for ai models and llm tooling. It runs as a hosted remote server, so there is nothing to install locally.
Nonobench measures how well language models solve nonogram (Picross) puzzles, using grids from 5x5 up to 20x20. Its public site ranks dozens of model variants by solve rate with 95% confidence intervals, and also shows total cost, total run time, a hard mode and an effort comparison that contrasts low and high reasoning settings.
The registry lists a streamable-http endpoint at nonobench.com/mcp so an MCP client can reach the benchmark directly. The listing does not name individual tools, so what a client can query is best checked by connecting to the endpoint.
The benchmark itself is open source. A Bun-based runner sends puzzles to models through an OpenRouter key, stores results in SQLite and never reruns a puzzle that already has a successful result. Answers are graded strictly: a grid counts only if it satisfies every clue. It can also test a model you host yourself through any OpenAI-compatible server such as vLLM, llama.cpp, LM Studio or Ollama.
Install
Run in a terminal
claude mcp add --transport http nonobench https://www.nonobench.com/mcpAdd to Settings → Connectors → Add custom connector
https://www.nonobench.com/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"nonobench": {
"url": "https://www.nonobench.com/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"nonobench": {
"type": "http",
"url": "https://www.nonobench.com/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"nonobench": {
"serverUrl": "https://www.nonobench.com/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"nonobench": {
"type": "http",
"url": "https://www.nonobench.com/mcp"
}
}
}What it does
- Nonogram benchmark
- Scores LLMs on nonogram puzzles from 5x5 to 20x20 grids.
- Accuracy with intervals
- Shows solve rates with 95% confidence intervals per model.
- Cost and time tracking
- Reports total spend and run time for each model variant.
- Reasoning effort ladder
- Compares results at low and high reasoning effort levels.
- Local model support
- Runner can test models served through OpenAI-compatible endpoints.
- Append-only results
- A model and puzzle pair with a successful result is never overwritten.
Good for
- Compare reasoning models on a constraint-satisfaction task
- Check whether a higher reasoning effort setting pays off in accuracy per dollar
- Benchmark a self-hosted model against commercial ones
- Pull leaderboard data into an AI assistant for research notes
Pricing
Open source
Open-source benchmark; no prices listed
Prices change. Check nonobench.com before you buy.
Nonobench FAQ
Which grid sizes does it cover?
The benchmark spans 5x5 to 20x20 nonograms, with 20x20 as an opt-in extended tier for runs.
How are answers graded?
A grid is correct only if it satisfies every row and column clue.
Can I benchmark my own model?
Yes. The runner works with any server that exposes an OpenAI-compatible /v1 endpoint, including vLLM, llama.cpp, LM Studio and Ollama.
Is Nonobench free?
Yes. Nonobench is open source, so you can use and self-host it for free.
How do I add Nonobench to Claude Code?
Run this command in your terminal: claude mcp add --transport http nonobench https://www.nonobench.com/mcp
What are the best alternatives to Nonobench?
The closest alternatives to Nonobench are Agent-Native Analytics, Claude Code Ultimate Guide, Google Ads + Meta Ads + SEO/GEO MCP, Tapetide - Indian Stock Market MCP and World Monitor.
Alternatives to Nonobench

Agent-Native Analytics
MCP serverOpen-source analytics app where an AI agent builds charts and dashboards from your data
- Remote
- Open source
- 7.7K GitHub stars
- Analytics and data

Claude Code Ultimate Guide
MCP serverMCP server for searching the Claude Code Ultimate Guide from any MCP client
Runs locally- Open source
- 6.5K GitHub stars
- AI models and LLM tooling

Google Ads + Meta Ads + SEO/GEO MCP
MCP serverHosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat
- 8 tools
- Remote
- Freemium
- 8.1K GitHub stars
- Marketing and SEO

Tapetide - Indian Stock Market MCP
MCP serverIndian NSE and BSE stock market data for AI assistants through 48 MCP tools
- 17 tools
- Remote
- Freemium
- 150 GitHub stars
- Finance and payments

World Monitor
MCP serverLive geopolitical intelligence dashboard with an API and MCP server for agents
- Remote
- Freemium
- Analytics and data

CompanyData
MCP serverSearch and enrich company records from official registers via the CompanyData API
- 3 tools
- Remote
- Paid
- 3 GitHub stars
- Analytics and data