# Searching the Internet for Challenging Benchmarks at Scale

Source: [Searching the Internet for Challenging Benchmarks at Scale](<https://www.sota2.com/research/paper/68fa7bcc1e209131cea8a89f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-28T05:00:05.404Z

Authors: Wenda Xu, Vil\\'em Zouhar, Parker Riley, Mara Finkelstein, Markus Freitag, Daniel Deutsch

Published: 2025-09-30T17:55:47.000Z

## Abstract

Many static benchmarks are beginning to saturate: as models rapidly improve, they achieve near-perfect scores on fixed test sets, leaving little headroom to expose genuine model weaknesses -- and even expert-curated challenge sets quickly saturate after hillclimbing. We present a fully automatic framework that searches the Internet at scale to construct challenging benchmarks without human curation. The key insight is to model the Internet as a vast space of topics and formalize the search as a multi-armed bandit problem, where each topic's difficulty is revealed only through expensive sample-and-evaluate queries. Our epsilon-greedy strategy identifies the most challenging topics while exploring only 6% of the search space -- a 100 times cost reduction over exhaustive evaluation. We validate on machine translation and knowledge question answering, confirming that discovered difficulty is robust across independent metrics (GEMBA-SQA and MetricX), languages, and models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Machine Translation on Discovered Challenge Topics](<https://www.sota2.com/research/sota/machine-translation-on-discovered-challenge-topics.md>): Machine Translation; Discovered Challenge Topics; Difficulty Score: 21.

- [Machine Translation on WMT '24 (Difficulty and Word Count Metrics)](<https://www.sota2.com/research/sota/machine-translation-on-wmt-24-difficulty-and-word-count-metrics.md>): Machine Translation; WMT 24; Difficulty Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2509.26619>)

[PDF](<https://arxiv.org/pdf/2509.26619.pdf>)
