Iris
Score AI agent runs for quality, PII leaks, prompt injection and cost, on your own machine
- Pricing
- Open source
- Platforms
- self-hosted, CLI, macOS
- Runs
- Locally
- Version
- 0.21.0
- Listing updated
- 2 days ago
About Iris
Iris is an open-source MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.
Iris takes an agent run and returns a verdict instead of leaving quality to eyeballing a few prompts. Your agent calls log_trace, a host hook forwards each turn, or the iris-eval ingest command loads a trace file, and every run received is scored against 28 built-in rules. PII detection covers 21 patterns such as SSN, card numbers, IBAN, medical record numbers, API keys and seed phrases, and prompt injection checks use 38 patterns. Traces record hierarchical spans, per-tool latency, token usage and cost in USD, with budget thresholds per trace or agent.
Twelve tools cover logging, rule evaluation, an LLM-as-judge and semantic citation verification. The judge and citation check use your own Anthropic or OpenAI key, with a per-evaluation cost cap, and no traffic goes through the vendor. Data stays in local SQLite and a dashboard serves at localhost:6920; npx @iris-eval/mcp-server --demo seeds sample failures.
It needs Node.js 22.13 or later, installs for clients such as Claude Code, Cursor, Windsurf, Zed and Codex, and is MIT licensed. Nothing leaves your machine unless you enable an OpenTelemetry export or the LLM judge.
Install
Run in a terminal
claude mcp add mcp-server --env IRIS_API_KEY='<IRIS_API_KEY>' --env IRIS_ANTHROPIC_API_KEY='<IRIS_ANTHROPIC_API_KEY>' --env IRIS_OPENAI_API_KEY='<IRIS_OPENAI_API_KEY>' --env IRIS_WEBHOOK_SECRET='<IRIS_WEBHOOK_SECRET>' -- npx -y @iris-eval/mcp-serverAdd to claude_desktop_config.json
{
"mcpServers": {
"mcp-server": {
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_API_KEY": "<IRIS_API_KEY>",
"IRIS_ANTHROPIC_API_KEY": "<IRIS_ANTHROPIC_API_KEY>",
"IRIS_OPENAI_API_KEY": "<IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<IRIS_WEBHOOK_SECRET>"
}
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"mcp-server": {
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_API_KEY": "<IRIS_API_KEY>",
"IRIS_ANTHROPIC_API_KEY": "<IRIS_ANTHROPIC_API_KEY>",
"IRIS_OPENAI_API_KEY": "<IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<IRIS_WEBHOOK_SECRET>"
}
}
}
}Add to .vscode/mcp.json
{
"servers": {
"mcp-server": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_API_KEY": "<IRIS_API_KEY>",
"IRIS_ANTHROPIC_API_KEY": "<IRIS_ANTHROPIC_API_KEY>",
"IRIS_OPENAI_API_KEY": "<IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<IRIS_WEBHOOK_SECRET>"
}
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"mcp-server": {
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_API_KEY": "<IRIS_API_KEY>",
"IRIS_ANTHROPIC_API_KEY": "<IRIS_ANTHROPIC_API_KEY>",
"IRIS_OPENAI_API_KEY": "<IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<IRIS_WEBHOOK_SECRET>"
}
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"mcp-server": {
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_API_KEY": "<IRIS_API_KEY>",
"IRIS_ANTHROPIC_API_KEY": "<IRIS_ANTHROPIC_API_KEY>",
"IRIS_OPENAI_API_KEY": "<IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<IRIS_WEBHOOK_SECRET>"
}
}
}
}Configuration
IRIS_API_KEYAPI key for HTTP authentication. Required to bind the HTTP transport or the dashboard beyond loopback (0.0.0.0, a LAN address, a container): without it the server refuses to startIRIS_API_KEY_FILEPath to a file whose trimmed contents are the API key (the secret-file pattern Docker and Kubernetes mount). Set this or IRIS_API_KEY, not both; further keys and rotation are security.apiKeys in config.jsonIRIS_ALLOW_UNAUTHENTICATEDSet to 1 to run a non-loopback bind with NO API key on purpose (lifts the startup refusal; the network is then your boundary)IRIS_DB_PATHSQLite database pathIRIS_SQLITE_DRIVERWhich SQLite driver holds the database: native (better-sqlite3, the default) or node (Node's built-in node:sqlite, Node 22.13+). Unset: native, falling back to node when the native module cannot loadIRIS_SEARCH_BUDGET_MSHow long one trace search may read before it answers with the matches found so far, in milliseconds (50-60000, default 1000)IRIS_SEARCH_INDEXon (default) or off. off keeps no full-text index of the traces: writes are cheaper, and a search reads the traces within its time budgetIRIS_LOG_LEVELLog level: debug, info, warn, errorIRIS_HOMEDirectory for the database, custom rules and preferences (default ~/.iris)IRIS_ANTHROPIC_API_KEYEnables evaluate_with_llm_judge and verify_citations with an Anthropic model. Optional: the deterministic rules never need a keyIRIS_OPENAI_API_KEYEnables evaluate_with_llm_judge and verify_citations with an OpenAI model. Optional: the deterministic rules never need a keyIRIS_LLM_JUDGE_MAX_COST_USD_PER_EVALHard cap on the worst-case cost of one LLM judge call (default 0.25); max_cost_usd can lower it for a call, never raise itIRIS_RELEVANCE_JUDGE_MODELA priced judge model; answers_the_ask then asks it on every evaluation with input, sending that input and output to the model's provider, and gates on its verdict (off by default)IRIS_LLM_JUDGE_DAILY_BUDGET_USDWhat every judge call together may spend per UTC day, per tenant, in USD (default 1): the relevance judge, evaluate_with_llm_judge and verify_citations. Past it no judge call is madeIRIS_RELEVANCE_JUDGE_DAILY_BUDGET_USDThe name of IRIS_LLM_JUDGE_DAILY_BUDGET_USD before 0.21.0. Read when the new name is unset, and it then limits every judge callIRIS_RELEVANCE_JUDGE_MAX_CALLS_PER_REQUESTRelevance judge calls one request may make (default 20)IRIS_RELEVANCE_JUDGE_REDACTon (default) replaces the PII and credentials no_pii flags before the relevance judge sends input and output to its provider; off sends them as they areIRIS_CITATION_ALLOW_FETCHLet verify_citations fetch URLs (off by default); a tool argument can turn it off for a call, never onIRIS_DASHBOARDStart the dashboard alongside the server (same as --dashboard)IRIS_DASHBOARD_PORTDashboard port (default 6920); IRIS_PORT is the MCP HTTP transport port (default 3000)IRIS_PORTMCP HTTP transport port (default 3000); only used with --transport httpIRIS_OTEL_ENDPOINTOTLP/HTTP collector URL; when set, every stored trace is also exported as OpenTelemetry spans (best-effort, asynchronous)IRIS_CITATION_DOMAINSComma-separated domain allowlist for verify_citations fetches (only consulted when IRIS_CITATION_ALLOW_FETCH is on); a tool argument can narrow it, never add to itIRIS_CITATION_MAX_COST_USD_TOTALThe most one verify_citations call may spend on judge calls, in USD (default 1); max_cost_usd_total can lower it, never raise itIRIS_TRANSPORTTransport: stdio (default) or http; same as --transportIRIS_HOSTBind host for the HTTP transport (default 127.0.0.1); only used with --transport httpIRIS_ALLOWED_ORIGINSComma-separated Origin allowlist for the HTTP transport and dashboard CORS (default http://localhost:*)IRIS_MAX_INPUT_ELEMENTSThe most array elements and object keys one input may hold before it is validated (default 100000): a tool call's arguments, and a dashboard or OTLP JSON request body. A tool call over it gets an error result; a body, 413. OTLP protobuf has its own ceiling, one element per 5 bytes of the request size limitIRIS_MAX_TOOL_INPUT_ELEMENTSThe name of IRIS_MAX_INPUT_ELEMENTS before 0.21.0, read when the new name is unsetIRIS_DASHBOARD_HOSTBind host for the dashboard (default 127.0.0.1); same as --dashboard-hostIRIS_OTEL_HEADERSComma-separated key=value headers sent with every OTLP export (for example an auth token for your collector)IRIS_OTEL_SERVICE_NAMEservice.name resource attribute on exported spans (default iris-eval)IRIS_OTEL_TIMEOUT_MSTimeout in milliseconds for each OTLP export requestIRIS_WEBHOOK_URLThe receiver of the webhook that fires on a moment (docs/webhooks.md); merged over notify.webhook in config.jsonIRIS_WEBHOOK_SECRETThe signing key of that webhook (any string, or whsec_ + base64); the iris format refuses to run without one
Tools this server provides (3)
Taken from the project documentation; the running server may differ.
- log_trace
- Captures a full agent run with spans, latency, token usage and cost.
- evaluate_with_llm_judge
- Scores an output with an LLM judge using your own API key.
- verify_citations
- Checks that cited sources support the output.
What it does
- Rule-based scoring
- 28 built-in rules plus custom Zod rules for output quality and safety.
- PII and injection checks
- 21 PII patterns and 38 prompt-injection patterns.
- Cost tracking
- Per-trace cost in USD with budget thresholds.
- LLM judge
- Optional judging with your own Anthropic or OpenAI key and a cost cap.
- Local dashboard
- Trace visualization and failures view backed by SQLite.
- One-command install
- Writes config for ten MCP clients and pins the version.
Good for
- Catch a PII leak in agent output
- Flag an agent run that overspent
- Audit agent traces from a file
- Verify citations in a generated answer
Pricing
Open source
MIT licensed, self-hosted, no limits; no hosted plan
Prices change. Check iris-eval.com before you buy.
Iris FAQ
Is it free?
Yes. The core is MIT licensed with no usage limits or account, and no hosted version exists today.
What does it need?
Node.js 22.13 or later.
Where is data stored?
Locally in SQLite; nothing leaves your machine unless you enable OpenTelemetry export or the LLM judge.
Does the LLM judge route through the vendor?
No. It calls Anthropic or OpenAI directly with your own key.
Is Iris free?
Yes. Iris is open source, so you can use and self-host it for free.
How do I add Iris to Claude Code?
Run this command in your terminal: claude mcp add mcp-server --env IRIS_API_KEY='<IRIS_API_KEY>' --env IRIS_ANTHROPIC_API_KEY='<IRIS_ANTHROPIC_API_KEY>' --env IRIS_OPENAI_API_KEY='<IRIS_OPENAI_API_KEY>' --env IRIS_WEBHOOK_SECRET='<IRIS_WEBHOOK_SECRET>' -- npx -y @iris-eval/mcp-server
What are the best alternatives to Iris?
The closest alternatives to Iris are REA, Claude Code Ultimate Guide, Agent Roast, Midplane and Leash.
Alternatives to Iris

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Claude Code Ultimate Guide
MCP serverMCP server for searching the Claude Code Ultimate Guide from any MCP client
Runs locally- Open source
- 6.5K GitHub stars
- AI models and LLM tooling
Agent Roast
MCP serverOffline review that flags agent changes which weaken tests, assertions or CI
Runs locally- Open source
- 4 Hacker News points
- Developer tools

Midplane
MCP serverPostgres gateway for AI agents with table grants, column masking and approval for writes
Runs locally- Open source
- 5 GitHub stars
- Databases

Leash
MCP serverReview what your AI touched, approve risky actions and undo supported changes
- 1 tool
- Remote
- Free
- 1 GitHub star
- Security
Zero-log MCP server with 49 tools for AI guardrails, secret scanning and EU compliance
- 49 tools
- Remote
- Freemium
- Security