Calibration on MSciNLI
3.75Expected Calibration Error (ECE)CaliDistverb. (Sa.)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CaliDistverb. (Sa.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 3.75 | 20.79 | |
| CaliDist (Sa.)LLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 3.92 | 22.58 | |
| CaliDist (As.)LLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 4.01 | 21.83 | |
| CaliDist (Pr.)LLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 4.06 | 21.59 | |
| CaliDist (Pr.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 4.82 | 18.88 | |
| CaliDistverb. (As.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 4.89 | 21.09 | |
| VerbalizedLLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 4.91 | 21.66 | |
| CaliDistverb. (Pr.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Verbalized2026.06 | 5.04 | 21.27 | |
| CaliDist (As.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 5.46 | 20.44 | |
| CaliDist (Sa.)LLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 5.87 | 21.03 | |
| CaliDistverb. (Sa.)LLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 11.47 | 24.89 | |
| VerbalizedLLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 13.25 | 24.98 | |
| CaliDistverb. (As.)LLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 13.59 | 24.53 | |
| CaliDistverb. (Pr.)LLM Model=GPT-4O-MINI, Confidence Type=Verbalized2026.06 | 13.96 | 25.73 | |
| EntropyLLM Model=GPT-4O-MINI, Confidence Type=Consistency2026.06 | 25.23 | 29.36 | |
| FSDLLM Model=GEMINI-2.0-FLASH, Confidence Type=Consistency2026.06 | 28.09 | 29.59 | |
| EntropyLLM Model=GEMINI-2.0-FLASH, Confidence Type=Consistency2026.06 | 28.14 | 29.69 | |
| FSDLLM Model=GPT-4O-MINI, Confidence Type=Consistency2026.06 | 28.76 | 30.77 | |
| VanillaLLM Model=GEMINI-2.0-FLASH, Confidence Type=Log-Prob.2026.06 | 29.81 | 30.38 | |
| ConsistencyLLM Model=GEMINI-2.0-FLASH, Confidence Type=Consistency2026.06 | 30.28 | 30.75 | |
| ConsistencyLLM Model=GPT-4O-MINI, Confidence Type=Consistency2026.06 | 33.2 | 33.28 | |
| VanillaLLM Model=GPT-4O-MINI, Confidence Type=Log-Prob.2026.06 | 34.97 | 35.14 |