# Nonobench

> Nonobench is an open-source MCP server for ai models and llm tooling. Remote MCP endpoint for the Nonobench LLM nonogram puzzle benchmark. It runs as a hosted remote server, so there is nothing to install locally.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** Web, API
- **Website:** https://www.nonobench.com
- **Repository:** https://github.com/mauricekleine/nonobench
- **MCP Registry name:** io.github.mauricekleine/nonobench
- **Categories:** AI models and LLM tooling, analytics and data
- **Listing updated:** 2026-11-16

## About

Nonobench measures how well language models solve nonogram (Picross) puzzles, using grids from 5x5 up to 20x20. Its public site ranks dozens of model variants by solve rate with 95% confidence intervals, and also shows total cost, total run time, a hard mode and an effort comparison that contrasts low and high reasoning settings.

The registry lists a streamable-http endpoint at nonobench.com/mcp so an MCP client can reach the benchmark directly. The listing does not name individual tools, so what a client can query is best checked by connecting to the endpoint.

The benchmark itself is open source. A Bun-based runner sends puzzles to models through an OpenRouter key, stores results in SQLite and never reruns a puzzle that already has a successful result. Answers are graded strictly: a grid counts only if it satisfies every clue. It can also test a model you host yourself through any OpenAI-compatible server such as vLLM, llama.cpp, LM Studio or Ollama.

## Features

- **Nonogram benchmark:** Scores LLMs on nonogram puzzles from 5x5 to 20x20 grids.
- **Accuracy with intervals:** Shows solve rates with 95% confidence intervals per model.
- **Cost and time tracking:** Reports total spend and run time for each model variant.
- **Reasoning effort ladder:** Compares results at low and high reasoning effort levels.
- **Local model support:** Runner can test models served through OpenAI-compatible endpoints.
- **Append-only results:** A model and puzzle pair with a successful result is never overwritten.

## Good for

- Compare reasoning models on a constraint-satisfaction task
- Check whether a higher reasoning effort setting pays off in accuracy per dollar
- Benchmark a self-hosted model against commercial ones
- Pull leaderboard data into an AI assistant for research notes

## Install

### Claude Code (Terminal)

```sh
claude mcp add --transport http nonobench https://www.nonobench.com/mcp
```

### Claude Desktop (Settings → Connectors → Add custom connector)

```sh
https://www.nonobench.com/mcp
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "nonobench": {
      "url": "https://www.nonobench.com/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "nonobench": {
      "type": "http",
      "url": "https://www.nonobench.com/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "nonobench": {
      "serverUrl": "https://www.nonobench.com/mcp"
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "nonobench": {
      "type": "http",
      "url": "https://www.nonobench.com/mcp"
    }
  }
}
```

## FAQ

### Which grid sizes does it cover?

The benchmark spans 5x5 to 20x20 nonograms, with 20x20 as an opt-in extended tier for runs.

### How are answers graded?

A grid is correct only if it satisfies every row and column clue.

### Can I benchmark my own model?

Yes. The runner works with any server that exposes an OpenAI-compatible /v1 endpoint, including vLLM, llama.cpp, LM Studio and Ollama.

### Is Nonobench free?

Yes. Nonobench is open source, so you can use and self-host it for free.

### How do I add Nonobench to Claude Code?

Run this command in your terminal: claude mcp add --transport http nonobench https://www.nonobench.com/mcp

### What are the best alternatives to Nonobench?

The closest alternatives to Nonobench are Agent-Native Analytics, Claude Code Ultimate Guide, Google Ads + Meta Ads + SEO/GEO MCP, Tapetide - Indian Stock Market MCP and World Monitor.

## Alternatives

- [Agent-Native Analytics](https://ai-powered.com/mcp/agent-native-analytics): Agent-Native Analytics is an open-source MCP server for analytics and data.
- [Claude Code Ultimate Guide](https://ai-powered.com/mcp/claude-code-ultimate-guide): Claude Code Ultimate Guide is an open-source MCP server for ai models and llm tooling.
- [Google Ads + Meta Ads + SEO/GEO MCP](https://ai-powered.com/mcp/google-ads-meta-ads-seo-geo-mcp): Google Ads + Meta Ads + SEO/GEO MCP is a freemium MCP server for marketing and seo.
- [Tapetide - Indian Stock Market MCP](https://ai-powered.com/mcp/tapetide-indian-stock-market-mcp): Tapetide - Indian Stock Market MCP is a freemium MCP server for finance and payments.
- [World Monitor](https://ai-powered.com/mcp/world-monitor): World Monitor is a freemium MCP server for analytics and data.
- [CompanyData](https://ai-powered.com/mcp/companydata): CompanyData is a paid MCP server for analytics and data.
- [ChartAnt](https://ai-powered.com/mcp/chartant): ChartAnt is a freemium MCP server for design and media.
- [Quantivex Research](https://ai-powered.com/mcp/quantivex-research): Quantivex Research is a free MCP server for finance and payments.

---
Source: https://ai-powered.com/mcp/nonobench
