Rag eval vs Rag perf
Rag eval is an open-source agent skill for knowledge and memory. Rag perf is an open-source agent skill for knowledge and memory.
- Rag eval is best for benchmark answer quality of my rag pipeline on my own documents.
- Rag perf is best for measure latency and throughput of a deployed rag server under load.
- What it is
- Rag eval is an open-source agent skill for knowledge and memory.Rag perf is an open-source agent skill for knowledge and memory.
- Best for
- Benchmark answer quality of my RAG pipeline on my own documentsMeasure latency and throughput of a deployed RAG server under load
- Kind
- SkillSkill
- Pricing
- Open sourceOpen source
- Platforms
- Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, GooseClaude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- GitHub stars
- 4K4K
- Last commit
- Nov 2026Nov 2026
- License
- Apache-2.0Apache-2.0
- Open source
- Good for
- Benchmark answer quality of my RAG pipeline on my own documents; Convert a CSV of questions and answers into a train.json dataset; Compare RAG answer scores with reranking turned on versus offMeasure latency and throughput of a deployed RAG server under load; Find the concurrency level where a RAG pipeline stops scaling; Compare retrieval or reranker top-k settings on profiling time
What Rag eval does
Filesystem dataset preparation
Explains the corpus and train.json layout and how to convert CSV, JSONL and public web sources into documents.
RAGAS quality evaluation
Runs the evaluate_rag.py driver to score answers with RAGAS metrics against a RAG server.
Retrieval and generation tuning
Covers flags for top_k, reranking, query rewriting and temperature when comparing configurations.
Result analysis
Points to scripts and summary JSON files for reviewing headline RAGAS metrics.
Error triage guidance
Lists common failure signals such as empty contexts, stream errors and ingestor URL mistakes with fixes.
What Rag perf does
Server profiling pass
Times each retrieval and generation stage of the deployed RAG server and infers the likely bottleneck.
aiperf load testing
Runs a concurrent load test measuring time to first token, latency and throughput through a bundled endpoint plugin.
YAML-driven presets
Three presets cover profile-only checks, a single benchmark point and multi-axis concurrency sweeps.
Synthetic query generation
Can generate benchmark queries through an OpenAI-compatible chat endpoint when no query file is supplied.
Unified benchmark reports
Writes per-point and aggregate Markdown, JSON and CSV reports with a side-by-side comparison table.
Result interpretation guidance
Explains how to read artifacts, compute scaling efficiency and flag zero citations or error spikes.
Rag eval vs Rag perf: questions
Is Rag eval or Rag perf cheaper?
Both are open source. Compare current plans on each official site.
What is the difference between Rag eval and Rag perf?
Rag eval is an open-source agent skill for knowledge and memory. Rag perf is an open-source agent skill for knowledge and memory.
