Skip to content
Tool

Werkbank

Self-hosted benchmark that scores local language models as agents on 40 tool-use tasks

Pricing
Open source
Platforms
Linux, Web, self-hosted
Listing updated
today

Similar tools

About Werkbank

Werkbank is an open-source AI tool for developer tools. It is available on Linux, Web, self-hosted and CLI.

The README says Werkbank is a benchmark you run against a language model you serve yourself. It sends 40 fixed tasks through any OpenAI-compatible chat-completions endpoint, with tools available to the model, then checks each outcome automatically and records the time and tokens each solved task took.

Each task pairs a prompt and a set of tools with a deterministic environment. Code the model writes runs only inside containers. Tasks cover agentic multi-step jobs, coding repairs, security cases with injected instructions, retrieval, reliability, math and data analysis, and results are reported per setup rather than as a leaderboard.

It suits people who run models on their own hardware and want to see how a specific combination of model, quantization, server, settings and hardware performs as an agent. A guided start walks through server detection, mode choice and a three-pass series that ends in a local HTML result page.

What it does

Fixed task suite
Forty tasks in seven weighted categories, each with an automatic outcome check.
Guided setup
Detects a local server, runs a short preflight check and picks Trial, Quick, Normal or Full modes.
Containerised code execution
Model-written code runs only inside Docker containers.
Multi-pass scoring
Runs three seeded passes and reports solved tasks, intervals and subgoal fulfilment.
Offline result page
Produces a script-free HTML result page that opens in any browser.
Browser interface
Optional local page for starting series, watching progress and viewing results.

Good for

  • Compare quantizations or serving settings of a local model on agent tasks
  • Check whether a self-hosted model handles tool calls and token counts correctly before use
  • Measure how long and how many tokens a local agent setup needs per solved task
  • Produce a reproducible local result page for a specific hardware and model setup

Pricing

Open source

Apache License 2.0 for code; documentation under CC BY 4.0.

Werkbank FAQ

Which model servers does it work with?

The README lists vLLM, llama.cpp, SGLang, Ollama and LM Studio, provided the endpoint is OpenAI-compatible and returns native tool calls and token counts.

What platforms are supported?

Linux, or Linux inside WSL2 on Windows, with Python 3.12 or newer and a Docker Engine the user may run.

Does it publish a leaderboard?

No. The README says the result page ranks nothing and shows no numbers of other models.

Are results sent anywhere automatically?

Submission is optional. The werkbank submit command shows exactly what would be sent and sends nothing without the user's confirmation.

Is Werkbank free?

Yes. Werkbank is open source, so you can use and self-host it for free.

What are the best alternatives to Werkbank?

The closest alternatives to Werkbank are Tensward, PromptJury, Lynx, codesnippets and AI Time Estimator.

Where this comes from

  • Description: written from the project's README.