# A Step Towards Mixture of Grader: Statistical Analysis of Existing Automatic Evaluation Metrics

Source: [A Step Towards Mixture of Grader: Statistical Analysis of Existing Automatic Evaluation Metrics](<https://www.sota2.com/research/paper/68fa75301e209131cea663c3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:10.172Z

Authors: Yun Joon Soh, Jishen Zhao

Published: 2024-10-13T22:10:42.000Z

## Abstract

The explosion of open-sourced models and Question-Answering (QA) datasets emphasizes the importance of automated QA evaluation. We studied the statistics of the existing evaluation metrics for a better understanding of their limitations. By measuring the correlation coefficients of each evaluation metric concerning human-like evaluation score, we observed the following: (1) existing metrics have a high correlation among them concerning the question type (e.g., single word, single phrase, etc.), (2) no single metric can adequately estimate the human-like evaluation. As a potential solution, we discuss how a Mixture Of Grader could potentially improve the auto QA evaluator quality.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2410.10030>)

[PDF](<https://arxiv.org/pdf/2410.10030.pdf>)
