Confidence Calibration on LogiQA (out-of-distribution)
8ECEADVICE w/ ConfTuner
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ADVICE w/ ConfTunerModel=GEMMA2 9B-IT2025.10 | 8 | 6.2 | 24.6 | 69 | |
| ADVICEModel=GEMMA2 9B-IT2025.10 | 11.9 | 10.9 | 25.8 | 57.8 | |
| ConfTunerModel=GEMMA2 9B-IT2025.10 | 18.4 | 17.9 | 23.8 | 71.4 | |
| ADVICEModel=LLAMA3.1 8B INSTRUCT2025.10 | 23 | 21.2 | 30.1 | 57.9 | |
| ADVICE w/ ConfTunerModel=MISTRAL 7B INSTRUCT2025.10 | 24 | 23.1 | 28.1 | 68 | |
| ConfTunerModel=MISTRAL 7B INSTRUCT2025.10 | 24.8 | 24.8 | 22.1 | 63.1 | |
| ADVICE w/ ConfTunerModel=LLAMA3.1 8B INSTRUCT2025.10 | 26 | 23.8 | 31.1 | 58.6 | |
| ConfTunerModel=LLAMA3.1 8B INSTRUCT2025.10 | 28.6 | 28.2 | 32.5 | 54.4 | |
| ADVICEModel=MISTRAL 7B INSTRUCT2025.10 | 35.6 | 31.6 | 38.2 | 59.3 | |
| PromptingModel=GEMMA2 9B-IT2025.10 | 39 | 38.7 | 40.3 | 50.3 | |
| Self-ConsistencyModel=GEMMA2 9B-IT2025.10 | 39 | 38.8 | 41.8 | 44.8 | |
| DefaultModel=GEMMA2 9B-IT2025.10 | 39.1 | 39 | 40.4 | 50.9 | |
| Self-ConsistencyModel=LLAMA3.1 8B INSTRUCT2025.10 | 45.3 | 45.1 | 46.2 | 45.6 | |
| Self-ConsistencyModel=MISTRAL 7B INSTRUCT2025.10 | 45.7 | 45.4 | 43.2 | 59 | |
| PromptingModel=LLAMA3.1 8B INSTRUCT2025.10 | 48.1 | 47.4 | 48 | 50.4 | |
| PromptingModel=MISTRAL 7B INSTRUCT2025.10 | 49.2 | 48 | 49.2 | 51.4 | |
| DefaultModel=MISTRAL 7B INSTRUCT2025.10 | 51.8 | 50.9 | 51.3 | 52.1 | |
| DefaultModel=LLAMA3.1 8B INSTRUCT2025.10 | 53.8 | 53.3 | 52.9 | 50.5 |