Question Answering Calibration on OOD Evaluation (Squad, WQ, CWQ, MSQ, PopQA)
5Squad Calibration ScoreThermometer
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ThermometerBackbone=Qwen2.5-7B-Instruct2025.10 | 5 | 5 | 1 | 6 | 10 | 7 | |
| DACABackbone=Qwen2.5-7B-Instruct2025.10 | 9 | 8 | 6 | 6 | 15 | 11 | |
| EliCal(1k)Backbone=Qwen2.5-7B-Instruct, LoRA rank=8, LoRA alpha=16, Batch size=162025.10 | 10 | 6 | 5 | 8 | 2 | 6 | |
| Consis-LexBackbone=Qwen2.5-7B-Instruct2025.10 | 14 | 7 | 13 | 32 | 26 | 20 | |
| Cal-Only(1k)Backbone=Qwen2.5-7B-Instruct, LoRA rank=8, LoRA alpha=16, Batch size=162025.10 | 17 | 15 | 16 | 12 | 7 | 12 | |
| ProbBackbone=Qwen2.5-7B-Instruct2025.10 | 31 | 54 | 33 | 13 | 16 | 25 | |
| Consis-SemBackbone=Qwen2.5-7B-Instruct2025.10 | 31 | 15 | 16 | 21 | 29 | 27 | |
| ElicitationBackbone=Qwen2.5-7B-Instruct, LoRA rank=8, LoRA alpha=16, Batch size=162025.10 | 31 | 9 | 11 | 19 | 24 | 24 | |
| Verbal-10Backbone=Qwen2.5-7B-Instruct2025.10 | 35 | 32 | 36 | 46 | 36 | 36 | |
| Verbal-0Backbone=Qwen2.5-7B-Instruct2025.10 | 42 | 29 | 44 | 46 | 57 | 48 | |
| N-ProbBackbone=Qwen2.5-7B-Instruct2025.10 | 46 | 24 | 46 | 65 | 60 | 52 |