# Achieving Reliable Human Assessment of Open-Domain Dialogue Systems

Source: [Achieving Reliable Human Assessment of Open-Domain Dialogue Systems](<https://www.sota2.com/research/paper/68fa6bfd1e209131cea317ab>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-24T16:51:46.375Z

Authors: Tianbo Ji, Yvette Graham, Gareth J. F. Jones, Chenyang Lyu, Qun Liu

Published: 2022-03-11T13:08:39.000Z

## Abstract

Evaluation of open-domain dialogue systems is highly challenging and development of better techniques is highlighted time and again as desperately needed. Despite substantial efforts to carry out reliable live evaluation of systems in recent competitions, annotations have been abandoned and reported as too unreliable to yield sensible results. This is a serious problem since automatic metrics are not known to provide a good indication of what may or may not be a high-quality conversation. Answering the distress call of competitions that have emphasized the urgent need for better evaluation techniques in dialogue, we present the successful development of human evaluation that is highly reliable while still remaining feasible and low cost. Self-replication experiments reveal almost perfectly repeatable results with a correlation of $r=0.969$. Furthermore, due to the lack of appropriate methods of statistical significance testing, the likelihood of potential improvements to systems occurring due to chance is rarely taken into account in dialogue evaluation, and the evaluation we propose facilitates application of standard tests. Since we have developed a highly reliable evaluation method, new insights into system performance can be revealed. We therefore include a comparison of state-of-the-art models (i) with and without personas, to measure the contribution of personas to conversation quality, as well as (ii) prescribed versus freely chosen topics. Interestingly with respect to personas, results indicate that personas do not positively contribute to conversation quality as expected.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Dialogue Evaluation on Ice-breaker human evaluation 1.0 (test)](<https://www.sota2.com/research/sota/dialogue-evaluation-on-ice-breaker-human-evaluation-1-0-test.md>): Dialogue Evaluation; Ice-breaker human evaluation 1.0 (test); Overall Score: not recorded.

- [Open-domain Dialogue Evaluation on Free Run 2 1.0 (secondary data collection run)](<https://www.sota2.com/research/sota/open-domain-dialogue-evaluation-on-free-run-2-1-0-secondary-data-collection-run.md>): Open-domain Dialogue Evaluation; Free Run 2 1.0 (secondary data collection run); Overall Quality Score: not recorded.

- [Open-domain Dialogue Evaluation on Free run Mechanical Turk 1 (initial data collection run)](<https://www.sota2.com/research/sota/open-domain-dialogue-evaluation-on-free-run-mechanical-turk-1-initial-data-collection-run.md>): Open-domain Dialogue Evaluation; Free run Mechanical Turk 1 (initial data collection run); Overall Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2203.05899>)

[PDF](<https://arxiv.org/pdf/2203.05899.pdf>)

[Implementation](<https://github.com/TianboJi/Dialogue-Eval>)