LLM Preflight
Check LLM outputs against your own rules before swapping a model, prompt or provider
- Platforms
- CLI
- Runs
- Locally
- Version
- 2.25.8
- Listing updated
- 4 days ago
Similar MCP servers
About LLM Preflight
LLM Preflight is an MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.
The core idea is a contract: you describe the JSON shape, routing labels or exact text a feature depends on, then test responses against it. A config can include a response that must pass and a negative control that must fail, so the validator itself gets checked. Exit codes carry the decision: 0 for pass, 1 for fail, 2 for an invalid config and 3 for inconclusive.
Two extras go beyond validation. A source audit scans a repository for literal model IDs that are retired or retiring across Anthropic, OpenAI and selected Gemini models, and suggests a replacement. Dry runs preview provider requests with estimated cost before any call is made. A local mock provider lets you try everything without keys, though a mock run is always inconclusive.
The README is careful about claims: a passing run is evidence for your account, prompts and validators, not production approval. It publishes an observed comparison where 26 model routes ran eight support-routing cases each, with 200 of 208 checks passing. It runs as a CLI, GitHub Action or stdio MCP server and needs Python 3.10 or later, with no third-party runtime dependencies.
Install
Run in a terminal
claude mcp add llm-preflight -- uvx llm-preflightAdd to claude_desktop_config.json
{
"mcpServers": {
"llm-preflight": {
"command": "uvx",
"args": [
"llm-preflight"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"llm-preflight": {
"command": "uvx",
"args": [
"llm-preflight"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"llm-preflight": {
"type": "stdio",
"command": "uvx",
"args": [
"llm-preflight"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"llm-preflight": {
"command": "uvx",
"args": [
"llm-preflight"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"llm-preflight": {
"command": "uvx",
"args": [
"llm-preflight"
]
}
}
}Using several servers? Build one config file with the MCP config generator.
What it does
- Output contracts
- Define allowed labels, JSON shape or exact output and check replies against them.
- Negative controls
- Fixtures that must fail prove the validator catches bad replies.
- Retired model audit
- Scans source for model IDs that are retired or retiring and names replacements.
- Dry-run cost preview
- Shows provider requests and estimated cost before spending anything.
- CI exit codes
- Distinct codes for pass, fail, invalid config and inconclusive.
- Local mock provider
- Try the workflow offline without keys or provider calls.
Good for
- Gate a model upgrade in CI
- Find hard-coded retired model IDs in a repo
- Verify a support-routing prompt only returns allowed labels
- Compare cost and latency of several providers on one contract
- Keep evidence for a prompt change review
LLM Preflight FAQ
Do I need an API key to try it?
No. The quick start and the source audit work without a key; live checks need provider credentials.
Does a pass mean it is safe for production?
No. The README says a passing run is evidence for your account, prompts and validators, not production approval.
Which providers does the model audit cover?
Anthropic, OpenAI and selected Gemini models.
What does it need to run?
Python 3.10 or later; the README says there are no third-party runtime dependencies.
How can I use it?
As a CLI, a GitHub Action or a local MCP server.
How do I add LLM Preflight to Claude Code?
Run this command in your terminal: claude mcp add llm-preflight -- uvx llm-preflight
What are the best alternatives to LLM Preflight?
The closest alternatives to LLM Preflight are DecisionLint: Jev/Luna Lint, Promptster, GroundLens, LLM Prover and LLM Assure.
Where this comes from
- Description: written from the project's README.
- Install steps: from the official MCP Registry entry.
