# Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA

Source: [Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA](<https://www.sota2.com/research/paper/68fa74831e209131cea62127>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-11-13T05:01:22.196Z

Authors: Sher Badshah, Hassan Sajjad

Published: 2024-08-17T16:01:45.000Z

## Abstract

The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2408.09235>)

[PDF](<https://arxiv.org/pdf/2408.09235.pdf>)
