Score RAG answer quality and retrieval quality before rollout with Ragas

Measure whether a RAG change actually improved answers and retrieval, instead of guessing from a few spot checks.

Prerequisites

Python environment, Ragas package, model provider credentials, evaluation dataset or testset generation inputs, access to the target RAG workflow

Installation

Use the upstream install or setup path that matches your environment:

• pip install ragas
• pip install git+https://github.com/vibrantlabsai/ragas

Requirements and caveats from upstream:

• <a href="https://www.python.org/">
• <img alt="Made with Python" src="https://img.shields.io/badge/Made%20with-Python-1f425f.svg?color=purple">
• python

Basic usage or getting-started notes:

• <a href="#fire-quickstart">Quick start</a> |

• Clone a Complete Example Project

• ragas comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using DiscreteMetric:

• Source: https://github.com/vibrantlabsai/ragas

• Extracted from upstream docs: https://raw.githubusercontent.com/vibrantlabsai/ragas/HEAD/README.md

Documentation

• https://docs.ragas.io/en/stable/

Source

• Agent Skill Exchange