Leaklens vs Nerf Watch
Leaklens is a freemium MCP server for developer tools. Nerf Watch is a free MCP server for ai models and llm tooling.
- Nerf Watch is free, while Leaklens is freemium.
- Nerf Watch is open source; Leaklens is not.
- Leaklens is best for screen a swe-bench results file for leaked tasks before trusting a score.
- Nerf Watch is best for ask whether a model's quality dipped this week.
- What it is
- Leaklens is a freemium MCP server for developer tools.Nerf Watch is a free MCP server for ai models and llm tooling.
- Best for
- Screen a SWE-bench results file for leaked tasks before trusting a scoreAsk whether a model's quality dipped this week
- Kind
- MCP serverMCP server
- Pricing
- FreemiumFree
- Platforms
- API, WebAPI, Web
- Runs
- RemotelyRemotely
- Open source
- Good for
- Screen a SWE-bench results file for leaked tasks before trusting a score; Drop flagged tasks from an evaluation run; Cite leakage rates in an eval reportAsk whether a model's quality dipped this week; Compare Claude and GPT models on the same suite; Chart a model's score history in a dashboard
What Leaklens does
Per-task leakage tier
Each task gets tier A, tier C or none, plus a flag, clear, disputed or pending state.
Quoted evidence
Matching passages are shown with their position in the issue or comments.
Fix-commit dates
The upstream fix commit and its date are returned for cutoff comparison.
Batch checks
Pass 1 to 50 task ids to see which tasks in a run are flagged.
Citable summary
Flagged counts and precision come with a ready-to-cite string.
Open API and CSV
The same data is available over GET endpoints and a CSV export.
What Nerf Watch does
Latest scores
Returns each model's current score, change from the last run and 7-day baseline.
Model history
Returns one model's retained scores, oldest first, with suite version.
Trend labels
Scores within 2 points of the baseline read as steady.
Private suite
Test questions stay hidden to reduce training on them.
No key needed
Hosted, read-only, no account.
Speed probe
The site also shows daily response times by model.
Leaklens vs Nerf Watch: questions
Is Leaklens or Nerf Watch cheaper?
Nerf Watch is the cheaper option: it is free.
What is the difference between Leaklens and Nerf Watch?
Leaklens is a freemium MCP server for developer tools. Nerf Watch is a free MCP server for ai models and llm tooling. Nerf Watch is free, while Leaklens is freemium. Nerf Watch is open source; Leaklens is not.