Confidence Calibration on TriviaQA (in-domain)
3.4Expected Calibration Error (ECE)ADVICE w/ ConfTuner
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ADVICE w/ ConfTunerModel=GEMMA2 9B-IT2025.10 | 3.4 | 2 | 16 | 77.1 | |
| ConfTunerModel=LLAMA3.1 8B INSTRUCT2025.10 | 5.2 | 1.1 | 15.3 | 66.3 | |
| ConfTunerModel=GEMMA2 9B-IT2025.10 | 5.7 | 2.9 | 14.3 | 82.7 | |
| ADVICEModel=GEMMA2 9B-IT2025.10 | 6.2 | 5.1 | 16.3 | 77.4 | |
| ADVICE w/ ConfTunerModel=MISTRAL 7B INSTRUCT2025.10 | 6.9 | 4.7 | 17.8 | 76.4 | |
| ConfTunerModel=MISTRAL 7B INSTRUCT2025.10 | 8.1 | 7.5 | 16.3 | 80.3 | |
| ADVICE w/ ConfTunerModel=LLAMA3.1 8B INSTRUCT2025.10 | 9.4 | 8.7 | 15.1 | 77.9 | |
| ADVICEModel=LLAMA3.1 8B INSTRUCT2025.10 | 10.4 | 9.8 | 14.8 | 77 | |
| PromptingModel=LLAMA3.1 8B INSTRUCT2025.10 | 12.1 | 12 | 17.6 | 59.7 | |
| ADVICEModel=MISTRAL 7B INSTRUCT2025.10 | 14.5 | 8 | 20.6 | 74.7 | |
| Self-ConsistencyModel=LLAMA3.1 8B INSTRUCT2025.10 | 15.7 | 15.1 | 22.2 | 58.6 | |
| DefaultModel=LLAMA3.1 8B INSTRUCT2025.10 | 16.9 | 16.6 | 21.2 | 56.2 | |
| PromptingModel=GEMMA2 9B-IT2025.10 | 21.4 | 21.3 | 24.9 | 53.9 | |
| DefaultModel=GEMMA2 9B-IT2025.10 | 21.9 | 21.8 | 25.3 | 52.7 | |
| PromptingModel=MISTRAL 7B INSTRUCT2025.10 | 27.6 | 24.8 | 31 | 52.8 | |
| Self-ConsistencyModel=GEMMA2 9B-IT2025.10 | 28.5 | 28.2 | 28.5 | 65.2 | |
| Self-ConsistencyModel=MISTRAL 7B INSTRUCT2025.10 | 31.3 | 30.3 | 31.6 | 68.5 | |
| DefaultModel=MISTRAL 7B INSTRUCT2025.10 | 32.8 | 30.5 | 35.3 | 51.6 |