Phoenix CLI vs Phoenix evals
Phoenix CLI is an open-source agent skill for ai models and llm tooling. Phoenix evals is an open-source agent skill for ai models and llm tooling.
- Phoenix CLI is best for find the traces with errors from the last hour and summarise what failed.
- Phoenix evals is best for write a code evaluator that checks llm outputs for a required format.
- What it is
- Phoenix CLI is an open-source agent skill for ai models and llm tooling.Phoenix evals is an open-source agent skill for ai models and llm tooling.
- Best for
- Find the traces with errors from the last hour and summarise what failedWrite a code evaluator that checks LLM outputs for a required format
- Kind
- SkillSkill
- Pricing
- Open sourceOpen source
- Platforms
- Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, GooseClaude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- GitHub stars
- 41.3K41.3K
- Last commit
- Nov 2026Nov 2026
- License
- Apache-2.0Apache-2.0
- Open source
- Good for
- Find the traces with errors from the last hour and summarise what failed; Write review notes on sampled multi-turn agent sessions; Group observed LLM failures into categories with countsWrite a code evaluator that checks LLM outputs for a required format; Build an LLM judge to score answer faithfulness in a RAG pipeline; Create a dataset and run an experiment comparing two prompt versions
What Phoenix CLI does
Trace and span inspection
Lists, filters and fetches traces, spans and sessions, including errors, token counts and model attributes.
Annotations and notes
Adds scores, labels and free-text notes to traces, spans and sessions, and removes them by identifier.
Open and axial coding
Walks through note-taking on sampled traffic and grouping those notes into a failure taxonomy.
Dataset and experiment review
Reads dataset examples, experiment runs and their errors, and prompt text from Phoenix.
Profiles and auth
Switches between multiple Phoenix endpoints and checks connection and authentication status.
GraphQL queries
Runs ad-hoc GraphQL queries and introspects types for anything the commands do not cover.
What Phoenix evals does
Code and LLM evaluators
Guidance for writing deterministic code checks and LLM judge evaluators in Python and TypeScript.
Evaluator validation
Steps for checking evaluator accuracy against human labels before trusting its results.
Experiments and datasets
References for creating datasets, running experiments and generating synthetic test data.
Error analysis workflow
Methods for reviewing traces, categorizing failures and deciding what to evaluate.
RAG evaluation
Patterns for evaluating retrieval quality and faithfulness in retrieval-augmented systems.
Production monitoring
Guidance on sampling traces, guardrails and continuous evaluation in production.
Phoenix CLI vs Phoenix evals: questions
Is Phoenix CLI or Phoenix evals cheaper?
Both are open source. Compare current plans on each official site.
What is the difference between Phoenix CLI and Phoenix evals?
Phoenix CLI is an open-source agent skill for ai models and llm tooling. Phoenix evals is an open-source agent skill for ai models and llm tooling.
