Skip to content
MCP server

EvalForge Lite

Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated…

Runs locallyStatus and history
Install
Pricing
Open source
Runs
Locally
Version
1.0.1
Listing updated
5 days ago

About EvalForge Lite

EvalForge Lite is an open-source MCP server. It runs locally on your machine and is started by your AI client.

Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.

Install

Run in a terminal

claude mcp add evalforge-lite --env OPENROUTER_API_KEY='<OPENROUTER_API_KEY>' -- uvx evalforge-lite

Add to claude_desktop_config.json

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "<OPENROUTER_API_KEY>"
      }
    }
  }
}

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "<OPENROUTER_API_KEY>"
      }
    }
  }
}

Add to .vscode/mcp.json

{
  "servers": {
    "evalforge-lite": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "<OPENROUTER_API_KEY>"
      }
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "<OPENROUTER_API_KEY>"
      }
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ],
      "env": {
        "OPENROUTER_API_KEY": "<OPENROUTER_API_KEY>"
      }
    }
  }
}

Configuration

  • OPENROUTER_API_KEYOptional. OpenRouter API key held by the server so tool calls need no credentials for OpenRouter. If unset, pass creds in each tool call.

Pricing

Open source

EvalForge Lite FAQ

Is EvalForge Lite free?

Yes. EvalForge Lite is open source, so you can use and self-host it for free.

How do I add EvalForge Lite to Claude Code?

Run this command in your terminal: claude mcp add evalforge-lite --env OPENROUTER_API_KEY='<OPENROUTER_API_KEY>' -- uvx evalforge-lite