Calibration on HellaSwag
3.18Expected Calibration Error (ECE)CaliDist (As.)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CaliDist (As.)LLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 3.18 | 10.84 | |
| CaliDist (Pr.)LLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 3.36 | 10.48 | |
| CaliDist (Pr.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 4.9 | 8.27 | |
| CaliDist (Sa.)LLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 5.07 | 11.04 | |
| CaliDist (As.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 5.16 | 8.55 | |
| CaliDist (Sa.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 5.89 | 8.69 | |
| EntropyLLM Model=GEMINI-2.0-FLASH, Confidence Type=Consistency2026.06 | 8.76 | 9.16 | |
| FSDLLM Model=GEMINI-2.0-FLASH, Confidence Type=Consistency2026.06 | 8.79 | 9.11 | |
| ConsistencyLLM Model=GEMINI-2.0-FLASH, Confidence Type=Consistency2026.06 | 9.23 | 9.39 | |
| FSDLLM Model=GPT-4O-MINI, Confidence Type=Consistency2026.06 | 9.37 | 11.51 | |
| VanillaLLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 9.88 | 9.61 | |
| ConsistencyLLM Model=GPT-4O-MINI, Confidence Type=Consistency2026.06 | 10.9 | 11.99 | |
| EntropyLLM Model=GPT-4O-MINI, Confidence Type=Consistency2026.06 | 11.04 | 12.31 | |
| VanillaLLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 12.96 | 13.14 | |
| VerbalizedLLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 13.79 | 14.61 | |
| CaliDistverb. (Sa.)LLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 13.79 | 14.59 | |
| CaliDistverb. (As.)LLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 14.38 | 14.61 | |
| CaliDistverb. (Pr.)LLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 14.83 | 14.26 | |
| VerbalizedLLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 22.6 | 13.8 | |
| CaliDistverb. (As.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 22.6 | 13.8 | |
| CaliDistverb. (Pr.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 22.6 | 13.8 | |
| CaliDistverb. (Sa.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 22.6 | 13.8 |