# Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity

Source: [Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity](<https://www.sota2.com/research/paper/6a279de090966ade919bf502>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-10T05:00:46.690Z

Authors: Zifan Lyu, Chahine Nejma, Tobias Wegel, Fanny Yang, Florian E. Dorner

Published: 2026-06-05T17:03:19.000Z

## Abstract

Large Language Models are typically benchmarked by evaluating every model on every test query. For practitioners seeking the best model to deploy, this is often wasteful: if a model clearly performs worse than others, there is no need to precisely estimate its performance. Best-arm identification algorithms can be naturally applied to drastically reduce costs by adaptively allocating evaluation budget. Further, language models often respond similarly to the same prompt-a property previous work has tried to leverage with mixed success. We propose Synchronized Successive Rejects (SySRs), augmenting the classical Successive Rejects algorithm with paired comparisons. Unlike prior attempts to leverage model similarity in best-model identification, our approach is hyperparameter-free and enjoys performance guarantees that improve with the degree of similarity between evaluated models. Empirically, our method outperforms all baselines in terms of average error rate across 15 standard benchmarks, and in terms of worst-case budget for reliably identifying the best model.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Model Selection on 15 datasets (average)](<https://www.sota2.com/research/sota/model-selection-on-15-datasets-average.md>): Model Selection; 15 datasets (average); Identification Accuracy (8% Budget): 83.5.

- [Best Model Identification on 15 datasets aggregate (overall)](<https://www.sota2.com/research/sota/best-model-identification-on-15-datasets-aggregate-overall.md>): Best Model Identification; 15 datasets aggregate (overall); Mean 90% CB: 8.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.07726>)

[PDF](<https://arxiv.org/pdf/2606.07726.pdf>)