# Confidence Calibration on SimpleQA Source: [Confidence Calibration on SimpleQA]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:37:56.351Z Evaluation of model confidence calibration performance on the SimpleQA dataset, measured using metrics including SNR Gain, Conf AUC, Absolute Accuracy, Smooth ECE, Brier Score, NLL, and Prediction Accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Confidence Calibration | | Dataset (including any stated split) | SimpleQA | | Metric | Brier Score | | Unit | SCALAR | | Direction | Lower is better | | Results loaded | 27 of 27 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Brier Score](): Lower is better; unit: SCALAR. - [SNR Gain](): Higher is better; unit: SCALAR. - [Confidence AUC](): Higher is better; unit: SCALAR. - [Absolute Accuracy](): Higher is better; unit: PERCENT. - [Smooth ECE](): Lower is better; unit: SCALAR. - [NLL](): Lower is better; unit: SCALAR. - [Prediction Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Brier Score | Result date | Report paper | | --- | --- | --- | --- | | Probe (train on TriviaQA) | 0.0386 | Not recorded | 6993f5e19ebc499c974098b5 | | P(True) | 0.0419 | Not recorded | 6993f5e19ebc499c974098b5 | | Probe (train on TriviaQA) | 0.06 | Not recorded | 6993f5e19ebc499c974098b5 | | Probe (train on TriviaQA) | 0.0638 | Not recorded | 6993f5e19ebc499c974098b5 | | GPT-5 | 0.178 | Not recorded | 694b739553fdd5c361739b4e | | Verbalized confidence | 0.1957 | Not recorded | 6993f5e19ebc499c974098b5 | | Claude-sonnet-4.5 | 0.214 | Not recorded | 694b739553fdd5c361739b4e | | Verbalized confidence | 0.2676 | Not recorded | 6993f5e19ebc499c974098b5 | | Claude-sonnet-4.5 | 0.284 | Not recorded | 694b739553fdd5c361739b4e | | Uniform random baseline | 0.301 | Not recorded | 6993f5e19ebc499c974098b5 | | GLM-4.6 | 0.31 | Not recorded | 694b739553fdd5c361739b4e | | Uniform random baseline | 0.3109 | Not recorded | 6993f5e19ebc499c974098b5 | | Uniform random baseline | 0.3133 | Not recorded | 6993f5e19ebc499c974098b5 | | Grok-4 | 0.327 | Not recorded | 694b739553fdd5c361739b4e | | Confidence-Brier | 0.341 | Not recorded | 694b739553fdd5c361739b4e | | Gemini-2.5-pro | 0.436 | Not recorded | 694b739553fdd5c361739b4e | | Probe (train on GSM8K) | 0.4451 | Not recorded | 6993f5e19ebc499c974098b5 | | Verbalized confidence | 0.4736 | Not recorded | 6993f5e19ebc499c974098b5 | | Probe (train on MATH) | 0.4846 | Not recorded | 6993f5e19ebc499c974098b5 | | Critic Value | 0.525 | Not recorded | 694b739553fdd5c361739b4e | | Probe (train on GSM8K) | 0.5465 | Not recorded | 6993f5e19ebc499c974098b5 | | Probe (train on MATH) | 0.5871 | Not recorded | 6993f5e19ebc499c974098b5 | | P(True) | 0.6072 | Not recorded | 6993f5e19ebc499c974098b5 | | P(True) | 0.6151 | Not recorded | 6993f5e19ebc499c974098b5 | | Probe (train on GSM8K) | 0.7048 | Not recorded | 6993f5e19ebc499c974098b5 | | Qwen3-4B-Instruct | 0.762 | Not recorded | 694b739553fdd5c361739b4e | | Probe (train on MATH) | 0.8297 | Not recorded | 6993f5e19ebc499c974098b5 | ### Probe (train on TriviaQA) Brier Score: 0.0386. Result ID: 699a7ca75698665ec406207b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Olmo-3-7B-Instruct","Cost":"\< 1"}. This record does not assert independent reproduction. ### P(True) Brier Score: 0.0419. Result ID: 699a7ca75698665ec406207a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Olmo-3-7B-Instruct","Cost":"1"}. This record does not assert independent reproduction. ### Probe (train on TriviaQA) Brier Score: 0.06. Result ID: 699a7ca75698665ec4062087. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"gpt-oss-20b","Cost":"\< 1"}. This record does not assert independent reproduction. ### Probe (train on TriviaQA) Brier Score: 0.0638. Result ID: 699a7ca75698665ec4062081. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Qwen3-8B","Cost":"\< 1"}. This record does not assert independent reproduction. ### GPT-5 Brier Score: 0.178. Result ID: 699945b25698665ec4afd56a. [Reporting paper]() Recorded evaluation configuration: {"Zero-shot":true}. This record does not assert independent reproduction. ### Verbalized confidence Brier Score: 0.1957. Result ID: 699a7ca75698665ec4062085. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"gpt-oss-20b","Cost":"L"}. This record does not assert independent reproduction. ### Claude-sonnet-4.5 Brier Score: 0.214. Result ID: 699945b25698665ec4afd569. [Reporting paper]() Recorded evaluation configuration: {"Thinking":true,"Zero-shot":true}. This record does not assert independent reproduction. ### Verbalized confidence Brier Score: 0.2676. Result ID: 699a7ca75698665ec4062079. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Olmo-3-7B-Instruct","Cost":"L"}. This record does not assert independent reproduction. ### Claude-sonnet-4.5 Brier Score: 0.284. Result ID: 699945b25698665ec4afd566. [Reporting paper]() Recorded evaluation configuration: {"Thinking":false,"Zero-shot":true}. This record does not assert independent reproduction. ### Uniform random baseline Brier Score: 0.301. Result ID: 699a7ca75698665ec4062084. [Reporting paper]() Recorded evaluation configuration: {"LLM":"gpt-oss-20b","Cost":"N/A"}. This record does not assert independent reproduction. ### GLM-4.6 Brier Score: 0.31. Result ID: 699945b25698665ec4afd567. [Reporting paper]() Recorded evaluation configuration: {"Zero-shot":true}. This record does not assert independent reproduction. ### Uniform random baseline Brier Score: 0.3109. Result ID: 699a7ca75698665ec406207e. [Reporting paper]() Recorded evaluation configuration: {"LLM":"Qwen3-8B","Cost":"N/A"}. This record does not assert independent reproduction. ### Uniform random baseline Brier Score: 0.3133. Result ID: 699a7ca75698665ec4062078. [Reporting paper]() Recorded evaluation configuration: {"LLM":"Olmo-3-7B-Instruct","Cost":"N/A"}. This record does not assert independent reproduction. ### Grok-4 Brier Score: 0.327. Result ID: 699945b25698665ec4afd564. [Reporting paper]() Recorded evaluation configuration: {"Zero-shot":true}. This record does not assert independent reproduction. ### Confidence-Brier Brier Score: 0.341. Result ID: 699945b25698665ec4afd568. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen3-4B-Instruct","Training Mode":"confidence-Brier","Zero-shot":true}. This record does not assert independent reproduction. ### Gemini-2.5-pro Brier Score: 0.436. Result ID: 699945b25698665ec4afd562. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Zero-shot":true}. This record does not assert independent reproduction. ### Probe (train on GSM8K) Brier Score: 0.4451. Result ID: 699a7ca75698665ec4062082. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Qwen3-8B","Cost":"\< 1"}. This record does not assert independent reproduction. ### Verbalized confidence Brier Score: 0.4736. Result ID: 699a7ca75698665ec406207f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Qwen3-8B","Cost":"L"}. This record does not assert independent reproduction. ### Probe (train on MATH) Brier Score: 0.4846. Result ID: 699a7ca75698665ec406207d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Olmo-3-7B-Instruct","Cost":"\< 1"}. This record does not assert independent reproduction. ### Critic Value Brier Score: 0.525. Result ID: 699945b25698665ec4afd565. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Base Model":"Qwen3-4B-Instruct","Training Mode":"ppo-value","Zero-shot":true}. This record does not assert independent reproduction. ### Probe (train on GSM8K) Brier Score: 0.5465. Result ID: 699a7ca75698665ec406207c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Olmo-3-7B-Instruct","Cost":"\< 1"}. This record does not assert independent reproduction. ### Probe (train on MATH) Brier Score: 0.5871. Result ID: 699a7ca75698665ec4062089. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"gpt-oss-20b","Cost":"\< 1"}. This record does not assert independent reproduction. ### P(True) Brier Score: 0.6072. Result ID: 699a7ca75698665ec4062080. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Qwen3-8B","Cost":"1"}. This record does not assert independent reproduction. ### P(True) Brier Score: 0.6151. Result ID: 699a7ca75698665ec4062086. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"gpt-oss-20b","Cost":"L"}. This record does not assert independent reproduction. ### Probe (train on GSM8K) Brier Score: 0.7048. Result ID: 699a7ca75698665ec4062088. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"gpt-oss-20b","Cost":"\< 1"}. This record does not assert independent reproduction. ### Qwen3-4B-Instruct Brier Score: 0.762. Result ID: 699945b25698665ec4afd563. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Zero-shot":true}. This record does not assert independent reproduction. ### Probe (train on MATH) Brier Score: 0.8297. Result ID: 699a7ca75698665ec4062083. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"LLM":"Qwen3-8B","Cost":"\< 1"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.