Attest
Testing framework for AI agents that checks replies, tool calls and state changes in CI
- Platforms
- API, CLI, self-hosted
- Built with
- pytest, Vitest
- Listing updated
- today
Similar agents
About Attest
Attest is an agent framework for developer tools. It is available on API, CLI and self-hosted.
Attest is a Python and TypeScript testing framework that runs with pytest or Vitest. Instead of only grading the text an agent returns, it inspects the sequence of tool calls and the changes an agent makes to external state. The README describes starting with deterministic checks and adding an LLM judge only where those checks cannot express a requirement.
The workflow is built around recording and replay. The capture step records a run's trace and model exchanges, applies redaction and drafts contracts from it. Strict replay then runs the agent's real code against those recorded model responses through a local proxy, so matching requests make no live model calls. The docs note that this removes model variability but not other sources of flaky tests, such as tool dependencies.
The project is aimed at teams that ship agents with tool access and want regression tests that fail a pull request when, for example, a refund tool is called twice for the same order. Checks report pass, fail or not_evaluated, and a CI policy can block merges on failures.
What it does
- Deterministic assertion layers
- Checks on output schema, required tools and substrings run without an API key and at no cost.
- Tool call contracts
- Behavior rules are written over the trace's tool calls, such as limiting how often a tool runs per order.
- Strict replay
- Replays recorded model responses through a local proxy so the agent's code runs without live model calls.
- CI merge gate
- A policy check decides whether failed or not-evaluated tests block a merge, and can be required in branch protection.
- Capture from incidents
- Records a run's trace and model exchanges, applies redaction, and drafts contracts for review.
Good for
- Adding regression tests for tool-call mistakes such as duplicate refunds
- Blocking agent changes in CI when recorded behavior contracts fail
- Testing agent code without spending on live model calls on every run
- Checking multi-step agent behavior with pytest or Vitest
Attest FAQ
Which languages and test runners does it support?
The README says it supports Python with pytest and TypeScript with Vitest.
Does it need an API key to run the example tests?
The README says the Python example's assertions are deterministic and need no API key, and the TypeScript test makes no provider calls.
Which agent frameworks are covered?
The README lists adapters including OpenAI, Anthropic, Gemini, Ollama, LangChain, CrewAI, OpenTelemetry, Google ADK and LlamaIndex, with maturity tiers that vary by language.
What are the best alternatives to Attest?
The closest alternatives to Attest are Kane CLI By TestMu AI, pydantic-ai, VSCode Extension, webqa-agent and harness-sdk.
Where this comes from
- Description: written from the project's website.

