Confidence Calibration on SciQ (test)
1.5ECESelf-Consistency
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Self-ConsistencyModel=LLAMA3.1 8B INSTRUCT2025.10 | 1.5 | 1.3 | 7.1 | 74 | |
| DefaultModel=LLAMA3.1 8B INSTRUCT2025.10 | 3.7 | 2.1 | 8.2 | 52 | |
| Self-ConsistencyModel=GEMMA2 9B-IT2025.10 | 4 | 4 | 6.3 | 70.3 | |
| PromptingModel=LLAMA3.1 8B INSTRUCT2025.10 | 4.3 | 0.7 | 7.2 | 51.4 | |
| DefaultModel=GEMMA2 9B-IT2025.10 | 4.9 | 1.5 | 5.2 | 50.3 | |
| PromptingModel=GEMMA2 9B-IT2025.10 | 5.1 | 1.4 | 5.2 | 50 | |
| ConfTunerModel=GEMMA2 9B-IT2025.10 | 6.8 | 2.9 | 6.9 | 77.2 | |
| ADVICEModel=LLAMA3.1 8B INSTRUCT2025.10 | 7.9 | 7.7 | 6.6 | 73.4 | |
| ADVICEModel=MISTRAL 7B INSTRUCT2025.10 | 8.3 | 8.1 | 12.5 | 63 | |
| ConfTunerModel=LLAMA3.1 8B INSTRUCT2025.10 | 9.5 | 9.1 | 9.5 | 58.4 | |
| PromptingModel=MISTRAL 7B INSTRUCT2025.10 | 9.8 | 8.5 | 14.9 | 51.7 | |
| DefaultModel=MISTRAL 7B INSTRUCT2025.10 | 10.3 | 9 | 15.3 | 50.3 | |
| ConfTunerModel=MISTRAL 7B INSTRUCT2025.10 | 10.8 | 9.9 | 14 | 68.6 | |
| ADVICEModel=GEMMA2 9B-IT2025.10 | 14.2 | 14.2 | 7.4 | 72.6 | |
| Self-ConsistencyModel=MISTRAL 7B INSTRUCT2025.10 | 14.8 | 14.5 | 16.9 | 67.1 |