# A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering Source: [A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering]() Collection index: [llms.txt]() Data updated: 2026-05-13T05:00:05.563Z Authors: Zhanliang Wang, Jiancong Xiao, Ruochen Jin, Shu Yang, Bojian Hou, Li Shen Published: 2026-05-08T19:53:49.000Z ## Abstract Calibration measures whether a model's predicted confidence aligns with its empirical accuracy, and is central to the reliable deployment of large language models (LLMs) in high-stakes domains such as medicine and law. While much recent work focuses on improving LLM calibration, the equally important question of how to evaluate it in realistic settings remains underdeveloped. Open-ended question answering (QA), the most common deployment setting for modern LLMs, is where existing evaluation methods fall short: logit-based metrics need restricted output formats and internal probabilities; verbalized confidence is self-reported and often overconfident; and sampling-based methods rely on task-specific extraction rules without a clear finite-sample target. We introduce Sem-ECE (Semantic-Sampling Expected Calibration Error), a calibration evaluation framework for open-ended QA that samples answers from the model, groups them into semantic classes, and uses the resulting frequencies as confidence. We study two estimators within this framework: Sem$\_1$-ECE, the same-sample self-consistency score, and Sem$\_2$-ECE, a held-out variant that separates answer selection from confidence evaluation. We prove both are asymptotically unbiased, and further show that they agree on easy questions but diverge on hard ones with Sem$\_2$ achieving strictly smaller calibration error, so their gap also serves as a diagnostic for question difficulty. Experiments on three open-ended QA benchmarks across five leading commercial LLMs match our theoretical predictions and show that Sem-ECE outperforms verbalized confidence and existing sampling-based methods, while complementing logit-based evaluation when internal probabilities are unavailable. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Open-ended Question Answering on SimpleQA](): Open-ended Question Answering; SimpleQA; Accuracy: not recorded. - [Open-ended Question Answering on HLE](): Open-ended Question Answering; HLE; Accuracy: not recorded. - [Open-ended Question Answering on PopQA](): Open-ended Question Answering; PopQA; Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()