Skip to content

Phoenix CLI vs Phoenix evals

Phoenix CLI is an open-source agent skill for ai models and llm tooling. Phoenix evals is an open-source agent skill for ai models and llm tooling.

  • Phoenix CLI is best for find the traces with errors from the last hour and summarise what failed.
  • Phoenix evals is best for write a code evaluator that checks llm outputs for a required format.
What it is
Phoenix CLI is an open-source agent skill for ai models and llm tooling.
Phoenix evals is an open-source agent skill for ai models and llm tooling.
Best for
Find the traces with errors from the last hour and summarise what failed
Write a code evaluator that checks LLM outputs for a required format
Kind
Skill
Skill
Pricing
Open source
Open source
Platforms
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
GitHub stars
41.3K
41.3K
Last commit
Nov 2026
Nov 2026
License
Apache-2.0
Apache-2.0
Open source
Good for
Find the traces with errors from the last hour and summarise what failed; Write review notes on sampled multi-turn agent sessions; Group observed LLM failures into categories with counts
Write a code evaluator that checks LLM outputs for a required format; Build an LLM judge to score answer faithfulness in a RAG pipeline; Create a dataset and run an experiment comparing two prompt versions

What Phoenix CLI does

  • Trace and span inspection

    Lists, filters and fetches traces, spans and sessions, including errors, token counts and model attributes.

  • Annotations and notes

    Adds scores, labels and free-text notes to traces, spans and sessions, and removes them by identifier.

  • Open and axial coding

    Walks through note-taking on sampled traffic and grouping those notes into a failure taxonomy.

  • Dataset and experiment review

    Reads dataset examples, experiment runs and their errors, and prompt text from Phoenix.

  • Profiles and auth

    Switches between multiple Phoenix endpoints and checks connection and authentication status.

  • GraphQL queries

    Runs ad-hoc GraphQL queries and introspects types for anything the commands do not cover.

What Phoenix evals does

  • Code and LLM evaluators

    Guidance for writing deterministic code checks and LLM judge evaluators in Python and TypeScript.

  • Evaluator validation

    Steps for checking evaluator accuracy against human labels before trusting its results.

  • Experiments and datasets

    References for creating datasets, running experiments and generating synthetic test data.

  • Error analysis workflow

    Methods for reviewing traces, categorizing failures and deciding what to evaluate.

  • RAG evaluation

    Patterns for evaluating retrieval quality and faithfulness in retrieval-augmented systems.

  • Production monitoring

    Guidance on sampling traces, guardrails and continuous evaluation in production.

Phoenix CLI vs Phoenix evals: questions

Is Phoenix CLI or Phoenix evals cheaper?

Both are open source. Compare current plans on each official site.

What is the difference between Phoenix CLI and Phoenix evals?

Phoenix CLI is an open-source agent skill for ai models and llm tooling. Phoenix evals is an open-source agent skill for ai models and llm tooling.

Keep comparing