Skip to content
AgentOpen-source agent

ARIA Evaluator

Open-source judge-panel evaluation for testing conversational AI agents before release

Pricing
Open source
Platforms
CLI, self-hosted, API
Listing updated
today

Similar agents

About ARIA Evaluator

ARIA Evaluator is an open-source agent for coding. It is available on CLI, self-hosted and API.

The README describes ARIA as a way to score conversational agents with several independent AI judges rather than one model. Each transcript is rated across 15 dimensions covering response quality, task completion, safety, customer experience and escalation. Where judges disagree, the case is routed to a person for a decision, and each result comes with its reasoning.

The project connects to existing agent platforms through adapters, including Amazon Connect, Amazon Lex, Azure Bot Service, Microsoft Copilot Studio, and generic OpenAPI, HTTP or WebSocket endpoints. Scenarios are defined in YAML and the judges run on Bedrock/Claude credentials supplied by the user. Adversarial scenarios cover prompt injection, jailbreaks and social engineering.

It suits teams building voice or chat agents in regulated settings who want an audit trail of evaluation results. Because the repository is Apache-2.0 and runs locally or on the user's own infrastructure, it is most relevant to teams comfortable operating their own evaluation stack.

What it does

Multi-judge scoring panel
Each conversation is reviewed by several independent AI judges so that a single model's bias does not decide the result.
15-dimension evaluation
Scores cover correctness, goal success, guardrail compliance, tone, escalation and vulnerability detection.
Adversarial scenario testing
Built-in scenarios probe prompt injection, jailbreak and social-engineering attempts against an agent.
Human review of disagreements
Results where judges conflict are sent to a human reviewer for the final decision.
Pluggable agent adapters
Adapters connect to Amazon Connect, Lex, Azure Bot Service, Copilot Studio and custom HTTP or WebSocket endpoints.
YAML-defined scenarios
Test scenarios are written as YAML files and run from the command line.

Good for

  • Testing a customer service voice bot on Amazon Connect before release
  • Checking an Azure or Copilot Studio agent for guardrail and injection weaknesses
  • Running regression evaluations after changing an agent's prompts or model
  • Producing per-dimension evaluation reports for compliance review

Pricing

Open source

Free open-source project under Apache-2.0; self-hosted.

Prices change. Check ariaeval.io before you buy.

ARIA Evaluator FAQ

Is ARIA Evaluator open source?

The README says the project is Apache-2.0 licensed on GitHub and can be self-hosted.

Which agent platforms can it test?

The site lists Amazon Connect, Amazon Lex V2, Azure Bot Service (Direct Line), Microsoft Copilot Studio agents, and any OpenAPI, HTTP or WebSocket endpoint.

Does it need a hosted account?

The quick start says no account or signup is required to run it locally.

Which model provider does it use for judges?

The README says it is configured with Bedrock/Claude credentials supplied by the user.

Is ARIA Evaluator free?

Yes. ARIA Evaluator is open source, so you can use and self-host it for free.

What are the best alternatives to ARIA Evaluator?

The closest alternatives to ARIA Evaluator are Agav, devika, Website, Auto-claude-code-research-in-sleep and Interview - founder about building Maige.

Where this comes from

  • Description: written from the project's website.