Ragas Documentation
by Ragas
Stop grading your RAG pipeline by vibes: about thirty metrics plus knowledge-graph test-set generation.
Overview
The Ragas docs are organized as Get Started, Core Concepts, How-to Guides and References, built around what the project calls an experiments-first approach: change one thing, re-run the same evaluation, compare the results. The metrics library is the substance. For retrieval-augmented generation it provides Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Response Relevancy and Faithfulness, plus multimodal faithfulness and relevance. For agents there are Topic Adherence, Tool Call Accuracy, Tool Call F1 and Agent Goal Accuracy. Natural-language comparison covers Factual Correctness, Semantic Similarity, non-LLM string similarity, BLEU, CHRF, ROUGE, String Presence and Exact Match; SQL adds Execution Based Datacompy Score and SQL Query Equivalence; general-purpose covers Aspect Critic, Simple Criteria Scoring and rubrics-based scoring; and a set of NVIDIA-contributed metrics adds Answer Accuracy, Context Relevance and Response Groundedness. Custom metrics are written with decorators. Test-set generation builds a knowledge graph from your documents using LLM-based and rule-based extractors plus relationship builders such as JaccardSimilarityBuilder, applies transforms in sequence with optional parallelism, then uses QuerySynthesizer to produce single-hop and multi-hop, specific and abstract queries across configurable personas and styles. It integrates with LangChain, LlamaIndex, LangSmith and Langfuse. The project is Apache-2.0 with 15.4k stars and 1.6k forks on GitHub, installs from PyPI, and offers a ragas quickstart command with templates including RAG evaluation.
At a Glance
- Topic
- RAG
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-4 hours for the quickstart and core concepts, then reference
- Provider
- Ragas
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Choose retrieval metrics such as context precision, recall and entities recall appropriately
- ✓Measure generation quality with faithfulness, response relevancy and noise sensitivity
- ✓Evaluate agents on topic adherence, tool call accuracy and goal accuracy
- ✓Generate a synthetic test set from your own documents via a knowledge graph
- ✓Control query difficulty with single-hop and multi-hop, specific and abstract synthesizers
- ✓Write custom metrics with decorators when the built-in library does not fit
- ✓Wire evaluation into LangChain, LlamaIndex, LangSmith or Langfuse workflows
Highlights
- •Roughly thirty metrics spanning RAG, agents, SQL and rubric scoring, so one framework covers the whole application rather than retrieval alone
- •Test-set generation is knowledge-graph based, producing genuine multi-hop questions instead of one shallow question per chunk
- •Mixes LLM-judged and deterministic metrics such as BLEU, ROUGE and exact match, so not every score costs an API call
- •Apache-2.0 with 15.4k stars and 1.6k forks, and the analytics collection is opt-out with its code public
- •Persona and style parameters let a generated test set reflect how your actual users phrase questions
Who It’s For
Best For
- ✓RAG teams replacing manual spot-checks with a repeatable evaluation suite
- ✓Engineers who need a labelled test set but have no annotation budget
- ✓Agent developers measuring tool-call correctness rather than only the final answer
- ✓Anyone comparing retrievers, chunk sizes or rerankers against the same yardstick
Prerequisites
- •Python plus an existing RAG or agent pipeline to evaluate
- •An LLM API key, since most Ragas metrics are model-judged
- •Understanding of precision and recall as evaluation concepts
FAQ
What is Ragas Documentation?
Documentation for Ragas, an open-source evaluation framework for LLM applications with roughly thirty built-in metrics covering RAG retrieval and generation, agent and tool-use behavior, SQL correctness and general rubric scoring. It also generates synthetic test sets from your own documents via a knowledge graph, and integrates with LangChain, LlamaIndex, LangSmith and Langfuse.
Is Ragas Documentation free?
Ragas Documentation is free to access.
What level is Ragas Documentation for?
Ragas Documentation is aimed at a intermediate audience. Recommended background: Python plus an existing RAG or agent pipeline to evaluate, An LLM API key, since most Ragas metrics are model-judged, Understanding of precision and recall as evaluation concepts.
How long does Ragas Documentation take?
Expect roughly ~3-4 hours for the quickstart and core concepts, then reference. Most learners work through it at their own pace.
What will I learn from Ragas Documentation?
You'll learn: Choose retrieval metrics such as context precision, recall and entities recall appropriately; Measure generation quality with faithfulness, response relevancy and noise sensitivity; Evaluate agents on topic adherence, tool call accuracy and goal accuracy; Generate a synthetic test set from your own documents via a knowledge graph; Control query difficulty with single-hop and multi-hop, specific and abstract synthesizers; Write custom metrics with decorators when the built-in library does not fit; Wire evaluation into LangChain, LlamaIndex, LangSmith or Langfuse workflows.
Topics
Sources
This page was written from 3 sources, 1 on domains other than docs.ragas.io.