# Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs Source: [Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs]() Collection index: [llms.txt]() Data updated: 2026-06-18T05:00:06.089Z Authors: Dong Huang, Jianbo Sun, Pengkun Yang Published: 2026-06-16T07:44:45.000Z ## Abstract Evaluating large language models (LLMs) is important for understanding their capabilities, comparing competing systems, and supporting the deployment of reliable models in practice. For open-ended tasks, pairwise evaluation has become a popular paradigm, in which two responses to the same prompt are compared and the resulting judgments are aggregated into an overall ranking. A central challenge of this paradigm is intransitivity: the induced comparison outcomes may fail to support any coherent global ranking. For example, one may observe cyclic preferences such as $A \\succ B \\succ C \\succ A$, or inconsistencies involving ties such as $A \\equiv B\\equiv C\\neq A$. Such contradictions make the resulting leaderboard unstable and challenging to interpret. In this paper, we propose a prompt perturbation framework for improving the consistency of pairwise LLM evaluation. Our approach generates perturbed variants of each prompt, uses the resulting comparison graphs to identify and filter out structurally inconsistent comparison patterns, and then applies standard ranking methods to the filtered comparisons. A key feature of the proposed framework is that graph-level structural consistency is incorporated explicitly into the evaluation pipeline before ranking aggregation. This provides a simple and principled way to reduce cyclic inconsistencies and improve the reliability of LLM rankings. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Code Generation on HumanEval (Normalized Spearman distance)](): Code Generation; HumanEval; Normalized Spearman Distance: 100. - [Mathematical Reasoning on MATH (Normalized Spearman distance)](): Mathematical Reasoning; MATH; Normalized Spearman Distance: 13. - [Model Ranking Agreement on MT-Bench Arena-derived reference ranking 1.0 (test)](): Model Ranking Agreement; MT-Bench Arena-derived reference ranking 1.0 (test); Coding Score: 79. ## Original sources and implementation [Original paper]() [PDF]()