# Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

Source: [Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations](<https://www.sota2.com/research/paper/6a4f2afa32fb937314875b26>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-10T05:00:01.278Z

Authors: Quentin Brabant

Published: 2026-07-08T11:44:28.000Z

## Abstract

This paper reports an empirical study evaluating the relevance of several RAG metrics. The experiment is based on a question-answering dataset created by human annotators from business data. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik). These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall. An analysis of correlations is conducted. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research. This paper is an English translation of a paper originally published in the French-speaking workshop EvalLLM (Brabant, 2026).

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.07302>)

[PDF](<https://arxiv.org/pdf/2607.07302.pdf>)
