Skip to content
MCP server

Nonobench

Remote MCP endpoint for the Nonobench LLM nonogram puzzle benchmark

Install
Pricing
Open source
Platforms
Web, API
Runs
Remotely
Version
1.1.0
Listing updated
1 day ago

About Nonobench

Nonobench is an open-source MCP server for ai models and llm tooling. It runs as a hosted remote server, so there is nothing to install locally.

Nonobench measures how well language models solve nonogram (Picross) puzzles, using grids from 5x5 up to 20x20. Its public site ranks dozens of model variants by solve rate with 95% confidence intervals, and also shows total cost, total run time, a hard mode and an effort comparison that contrasts low and high reasoning settings.

The registry lists a streamable-http endpoint at nonobench.com/mcp so an MCP client can reach the benchmark directly. The listing does not name individual tools, so what a client can query is best checked by connecting to the endpoint.

The benchmark itself is open source. A Bun-based runner sends puzzles to models through an OpenRouter key, stores results in SQLite and never reruns a puzzle that already has a successful result. Answers are graded strictly: a grid counts only if it satisfies every clue. It can also test a model you host yourself through any OpenAI-compatible server such as vLLM, llama.cpp, LM Studio or Ollama.

Install

Run in a terminal

claude mcp add --transport http nonobench https://www.nonobench.com/mcp

Add to Settings → Connectors → Add custom connector

https://www.nonobench.com/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "nonobench": {
      "url": "https://www.nonobench.com/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "nonobench": {
      "type": "http",
      "url": "https://www.nonobench.com/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "nonobench": {
      "serverUrl": "https://www.nonobench.com/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "nonobench": {
      "type": "http",
      "url": "https://www.nonobench.com/mcp"
    }
  }
}

What it does

Nonogram benchmark
Scores LLMs on nonogram puzzles from 5x5 to 20x20 grids.
Accuracy with intervals
Shows solve rates with 95% confidence intervals per model.
Cost and time tracking
Reports total spend and run time for each model variant.
Reasoning effort ladder
Compares results at low and high reasoning effort levels.
Local model support
Runner can test models served through OpenAI-compatible endpoints.
Append-only results
A model and puzzle pair with a successful result is never overwritten.

Good for

  • Compare reasoning models on a constraint-satisfaction task
  • Check whether a higher reasoning effort setting pays off in accuracy per dollar
  • Benchmark a self-hosted model against commercial ones
  • Pull leaderboard data into an AI assistant for research notes

Pricing

Open source

Open-source benchmark; no prices listed

Prices change. Check nonobench.com before you buy.

Nonobench FAQ

Which grid sizes does it cover?

The benchmark spans 5x5 to 20x20 nonograms, with 20x20 as an opt-in extended tier for runs.

How are answers graded?

A grid is correct only if it satisfies every row and column clue.

Can I benchmark my own model?

Yes. The runner works with any server that exposes an OpenAI-compatible /v1 endpoint, including vLLM, llama.cpp, LM Studio and Ollama.

Is Nonobench free?

Yes. Nonobench is open source, so you can use and self-host it for free.

How do I add Nonobench to Claude Code?

Run this command in your terminal: claude mcp add --transport http nonobench https://www.nonobench.com/mcp

What are the best alternatives to Nonobench?

The closest alternatives to Nonobench are Agent-Native Analytics, Claude Code Ultimate Guide, Google Ads + Meta Ads + SEO/GEO MCP, Tapetide - Indian Stock Market MCP and World Monitor.

Alternatives to Nonobench

  1. Open-source analytics app where an AI agent builds charts and dashboards from your data

    • Remote
    • Open source
    • 7.7K GitHub stars
    • Analytics and data
  2. MCP server for searching the Claude Code Ultimate Guide from any MCP client

    Runs locally
    • Open source
    • 6.5K GitHub stars
    • AI models and LLM tooling
  3. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  4. Indian NSE and BSE stock market data for AI assistants through 48 MCP tools

    • 17 tools
    • Remote
    • Freemium
    • 150 GitHub stars
    • Finance and payments
  5. World Monitor

    MCP server

    Live geopolitical intelligence dashboard with an API and MCP server for agents

    • Remote
    • Freemium
    • Analytics and data
  6. CompanyData

    MCP server

    Search and enrich company records from official registers via the CompanyData API

    • 3 tools
    • Remote
    • Paid
    • 3 GitHub stars
    • Analytics and data