Confidence Calibration on MMLU out-of-distribution
5.6ECEADVICE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ADVICEModel=GEMMA2 9B-IT2025.10 | 5.6 | 3.9 | 18.6 | 65.9 | |
| ADVICEModel=LLAMA3.1 8B INSTRUCT2025.10 | 8.6 | 7.6 | 20.7 | 69.2 | |
| ADVICE w/ ConfTunerModel=LLAMA3.1 8B INSTRUCT2025.10 | 9.6 | 7.1 | 20.9 | 68.7 | |
| ConfTunerModel=GEMMA2 9B-IT2025.10 | 11 | 7.3 | 20.2 | 75.7 | |
| ADVICE w/ ConfTunerModel=GEMMA2 9B-IT2025.10 | 11.7 | 11.7 | 19.9 | 66.8 | |
| ConfTunerModel=LLAMA3.1 8B INSTRUCT2025.10 | 13.9 | 13.9 | 24.2 | 58.2 | |
| DefaultModel=GEMMA2 9B-IT2025.10 | 21 | 21 | 24.7 | 50.1 | |
| PromptingModel=GEMMA2 9B-IT2025.10 | 21 | 21 | 24.6 | 50.4 | |
| ADVICE w/ ConfTunerModel=MISTRAL 7B INSTRUCT2025.10 | 21.8 | 21.3 | 26.5 | 69.9 | |
| Self-ConsistencyModel=GEMMA2 9B-IT2025.10 | 22.5 | 21.8 | 25.8 | 57.4 | |
| Self-ConsistencyModel=LLAMA3.1 8B INSTRUCT2025.10 | 25 | 25 | 29.9 | 53.9 | |
| PromptingModel=LLAMA3.1 8B INSTRUCT2025.10 | 25.5 | 25.3 | 28.5 | 52.8 | |
| DefaultModel=LLAMA3.1 8B INSTRUCT2025.10 | 26.9 | 26.7 | 29.7 | 50.8 | |
| ADVICEModel=MISTRAL 7B INSTRUCT2025.10 | 28.7 | 27.3 | 31.4 | 62.7 | |
| Self-ConsistencyModel=MISTRAL 7B INSTRUCT2025.10 | 34.5 | 34.3 | 36.1 | 61 | |
| DefaultModel=MISTRAL 7B INSTRUCT2025.10 | 35.3 | 35 | 37 | 51.5 | |
| ConfTunerModel=MISTRAL 7B INSTRUCT2025.10 | 36 | 36 | 34.3 | 70.8 | |
| PromptingModel=MISTRAL 7B INSTRUCT2025.10 | 36.3 | 34.1 | 38.2 | 49.9 |