Skip to content

Rag eval vs Rag perf

Rag eval is an open-source agent skill for knowledge and memory. Rag perf is an open-source agent skill for knowledge and memory.

  • Rag eval is best for benchmark answer quality of my rag pipeline on my own documents.
  • Rag perf is best for measure latency and throughput of a deployed rag server under load.
What it is
Rag eval is an open-source agent skill for knowledge and memory.
Rag perf is an open-source agent skill for knowledge and memory.
Best for
Benchmark answer quality of my RAG pipeline on my own documents
Measure latency and throughput of a deployed RAG server under load
Kind
Skill
Skill
Pricing
Open source
Open source
Platforms
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
GitHub stars
4K
4K
Last commit
Nov 2026
Nov 2026
License
Apache-2.0
Apache-2.0
Open source
Good for
Benchmark answer quality of my RAG pipeline on my own documents; Convert a CSV of questions and answers into a train.json dataset; Compare RAG answer scores with reranking turned on versus off
Measure latency and throughput of a deployed RAG server under load; Find the concurrency level where a RAG pipeline stops scaling; Compare retrieval or reranker top-k settings on profiling time

What Rag eval does

  • Filesystem dataset preparation

    Explains the corpus and train.json layout and how to convert CSV, JSONL and public web sources into documents.

  • RAGAS quality evaluation

    Runs the evaluate_rag.py driver to score answers with RAGAS metrics against a RAG server.

  • Retrieval and generation tuning

    Covers flags for top_k, reranking, query rewriting and temperature when comparing configurations.

  • Result analysis

    Points to scripts and summary JSON files for reviewing headline RAGAS metrics.

  • Error triage guidance

    Lists common failure signals such as empty contexts, stream errors and ingestor URL mistakes with fixes.

What Rag perf does

  • Server profiling pass

    Times each retrieval and generation stage of the deployed RAG server and infers the likely bottleneck.

  • aiperf load testing

    Runs a concurrent load test measuring time to first token, latency and throughput through a bundled endpoint plugin.

  • YAML-driven presets

    Three presets cover profile-only checks, a single benchmark point and multi-axis concurrency sweeps.

  • Synthetic query generation

    Can generate benchmark queries through an OpenAI-compatible chat endpoint when no query file is supplied.

  • Unified benchmark reports

    Writes per-point and aggregate Markdown, JSON and CSV reports with a side-by-side comparison table.

  • Result interpretation guidance

    Explains how to read artifacts, compute scaling efficiency and flag zero citations or error spikes.

Rag eval vs Rag perf: questions

Is Rag eval or Rag perf cheaper?

Both are open source. Compare current plans on each official site.

What is the difference between Rag eval and Rag perf?

Rag eval is an open-source agent skill for knowledge and memory. Rag perf is an open-source agent skill for knowledge and memory.

Keep comparing