Skip to content
MCP server

Leaklens

Check SWE-bench tasks for leaked fixes: tier, quoted evidence and fix-commit date

Status and history

Answered our latest tool-list check 21 hours ago with 2 tools. It will be marked verified after 3 successful checks in 7 days.

Pricing
Freemium
Platforms
API, Web
Runs
Remotely
Version
1.0.0
Listing updated
3 days ago

About Leaklens

Leaklens is a freemium MCP server for developer tools. It runs as a hosted remote server, so there is nothing to install locally.

Benchmark scores mean less if a model could have seen the answer. This server looks up SWE-bench Verified and Lite tasks and reports whether the issue or its comments state the fix (tier A) or contain lines of the gold patch (tier C), with the quoted passage and the date of the upstream fix commit to compare against a model's training cutoff.

Two read-only tools answer without a key or sign-up over a Streamable HTTP endpoint. One checks a single task or up to 50 at once, so a results file can be screened for flagged tasks. The other returns a benchmark's flagged counts, measured precision with its interval and a ready-to-cite string.

Every result carries a caveat that a fix dated before a cutoff shows exposure was possible, not that the model trained on it. The same data is available over plain GET endpoints and as a CSV, and a paid tier checks your own benchmark.

Install

Run in a terminal

claude mcp add --transport http leaklens https://leaklens.grain64.com/mcp

Add to Settings → Connectors → Add custom connector

https://leaklens.grain64.com/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "leaklens": {
      "url": "https://leaklens.grain64.com/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "leaklens": {
      "type": "http",
      "url": "https://leaklens.grain64.com/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "leaklens": {
      "serverUrl": "https://leaklens.grain64.com/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "leaklens": {
      "type": "http",
      "url": "https://leaklens.grain64.com/mcp"
    }
  }
}

Tools this server provides (2)

Read directly from the server.

check_task_leakage
Check whether coding tasks from SWE-bench-style benchmarks leaked their fix. Pass one instance ID for the full evidence, or up to 50 IDs in the queries field, as a JSON array of strings or one string separated by commas, spaces or new lines (a results file's task list), to get back which are flagged. Each task returns its tier, state and agreement, countedInHeadline, a one-clause why, a quoted sen
benchmark_leakage_summary
How leaky is a coding benchmark? Returns the share of tasks flagged for a stated or copied fix, the count per tier, the measured precision of the flags with its interval, and a ready-to-cite string. Optionally lists the flagged tasks of one repo (flaggedTasks, the headline's rule: Tier A where both checks agreed, plus Tier C) and the tasks flagged but counted apart (notCountedTasks: Tier B, Tier A

What it does

Per-task leakage tier
Each task gets tier A, tier C or none, plus a flag, clear, disputed or pending state.
Quoted evidence
Matching passages are shown with their position in the issue or comments.
Fix-commit dates
The upstream fix commit and its date are returned for cutoff comparison.
Batch checks
Pass 1 to 50 task ids to see which tasks in a run are flagged.
Citable summary
Flagged counts and precision come with a ready-to-cite string.
Open API and CSV
The same data is available over GET endpoints and a CSV export.

Good for

  • Screen a SWE-bench results file for leaked tasks before trusting a score
  • Drop flagged tasks from an evaluation run
  • Cite leakage rates in an eval report
  • Compare fix-commit dates with a model's training cutoff
  • Check one GitHub issue or pull request URL

Pricing

Freemium

Public lookups free. Own-benchmark checks: free 500 checks/month, Pro €49/month, audit pack €29 once.

Prices change. Check leaklens.grain64.com before you buy.

Leaklens FAQ

Do I need a key?

No. The MCP endpoint and lookups for SWE-bench Verified and Lite are free with no sign-up.

Which benchmarks are covered?

SWE-bench Verified and SWE-bench Lite.

Does a flag prove a model trained on the fix?

No. A fix commit dated before a cutoff shows exposure was possible, not that training happened.

Can I check my own benchmark?

Yes, on a free tier of 500 task checks a month, with a Pro plan at €49 a month and a €29 one-off audit pack.

Is Leaklens free?

Leaklens has a free plan, with paid plans for more usage or features.

How do I add Leaklens to Claude Code?

Run this command in your terminal: claude mcp add --transport http leaklens https://leaklens.grain64.com/mcp

What are the best alternatives to Leaklens?

The closest alternatives to Leaklens are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.

Alternatives to Leaklens

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. Codebase Memory

    MCP server

    Local MCP server that builds a code knowledge graph for AI coding agents

    Runs locally
    • Open source
    • 48.7K GitHub stars
    • Knowledge and memory
  3. GitHub

    MCP server

    Official MCP server connecting AI assistants to GitHub repos, issues and workflows

    Not checked yet
    • Remote
    • Open source
    • 34.5K GitHub stars
    • Developer tools
  4. MCP toolkit giving coding agents symbol-level code retrieval and editing

    Runs locally
    • Open source
    • 31.5K GitHub stars
    • Developer tools
  5. XERJ

    MCP server

    Local search engine for AI agents with cited answers over your code and docs

    Runs locally
    • Open source
    • 19.2K GitHub stars
    • Knowledge and memory
  6. Ouroboros

    MCP server

    Spec-first layer that pins requirements and verifies results of AI coding agent work

    Runs locally
    • Open source
    • 6.4K GitHub stars
    • Developer tools