falsepass
Test RL environment graders for wrong answers they pay and right ones they reject
- Pricing
- Open source
- Platforms
- CLI, self-hosted
- Runs
- Locally
- Version
- 0.15.0
- Listing updated
- 3 days ago
About falsepass
falsepass is an open-source MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.
A grader in an RL environment is effectively an assertion, and falsepass tests it like one: it calls the environment's own reward code on completions whose correct verdict is already known and reports every disagreement. Wrong answers that get paid are false passes, and right answers that are rejected are false fails. No model, rollouts or GPU are involved.
Probes include the reference answer with harmless variations; empty, echoed, every-option, hedged and wrong answers; malformed output; and repeats, plus a mutation search and a static scan of the reward source. Each failing probe becomes a stand-alone repro. The README describes one public environment where echoing the question back earned full reward because the grader only checked that the right answer appeared somewhere in the reply.
It works with verifiers (Environments Hub), OpenEnv, TRL reward functions, Inspect AI and HUD, and every audit runs in its own throw-away virtualenv (uv and Python 3.11+ needed). A GitHub Action can fail a build when a known-bad answer earns reward, and the MCP server exposes scan, init, audit, results, repro and discover.
Install
Run in a terminal
claude mcp add falsepass -- uvx falsepassAdd to claude_desktop_config.json
{
"mcpServers": {
"falsepass": {
"command": "uvx",
"args": [
"falsepass"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"falsepass": {
"command": "uvx",
"args": [
"falsepass"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"falsepass": {
"type": "stdio",
"command": "uvx",
"args": [
"falsepass"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"falsepass": {
"command": "uvx",
"args": [
"falsepass"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"falsepass": {
"command": "uvx",
"args": [
"falsepass"
]
}
}
}Tools this server provides (6)
Taken from the project documentation; the running server may differ.
- scan
- Static scan of reward code.
- init
- Sets up an environment for auditing.
- audit
- Probes the grader.
- results
- Returns headline numbers and failing probes.
- repro
- Creates a stand-alone repro for a probe.
- discover
- Finds environments to audit.
What it does
- False pass and fail probes
- Feeds graders completions with known verdicts and lists every mismatch.
- Mutation and static scan
- Searches variations and points at suspicious reward-code lines.
- Stand-alone repros
- Turns each failing probe into a reproducible script.
- Framework coverage
- Handles verifiers, OpenEnv, TRL, Inspect AI and HUD environments.
- CI gate
- A GitHub Action fails builds when a bad answer earns reward.
- Agent-ready output
- Non-interactive commands with JSON results and exit codes.
Good for
- Audit an RL environment before training on it
- Check that a reward function does not pay empty or echoed answers
- Gate pull requests that change a grader
- Have a coding agent produce reproduced leads for grader defects
Pricing
Open source
Free Python package
Prices change. Check super-eric.github.io before you buy.
falsepass FAQ
Does it need a model or GPU?
No. It runs the environment's own reward code on known completions.
What can it test?
verifiers, OpenEnv, TRL reward functions, Inspect AI and HUD graders.
How do I try it?
Run uvx falsepass demo, which audits a bundled toy environment.
What are the requirements?
uv on the PATH and Python 3.11 or newer.
Is falsepass free?
Yes. falsepass is open source, so you can use and self-host it for free.
How do I add falsepass to Claude Code?
Run this command in your terminal: claude mcp add falsepass -- uvx falsepass
What are the best alternatives to falsepass?
The closest alternatives to falsepass are REA, Codebase Memory, GitHub, Serena MCP: the IDE for your agent and XERJ.
Alternatives to falsepass

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Codebase Memory
MCP serverLocal MCP server that builds a code knowledge graph for AI coding agents
Runs locally- Open source
- 48.7K GitHub stars
- Knowledge and memory

GitHub
MCP serverOfficial MCP server connecting AI assistants to GitHub repos, issues and workflows
Not checked yet- Remote
- Open source
- 34.5K GitHub stars
- Developer tools

Serena MCP: the IDE for your agent
MCP serverMCP toolkit giving coding agents symbol-level code retrieval and editing
Runs locally- Open source
- 31.5K GitHub stars
- Developer tools

XERJ
MCP serverLocal search engine for AI agents with cited answers over your code and docs
Runs locally- Open source
- 19.2K GitHub stars
- Knowledge and memory

Ouroboros
MCP serverSpec-first layer that pins requirements and verifies results of AI coding agent work
Runs locally- Open source
- 6.4K GitHub stars
- Developer tools