Uncertainty Estimation (Factual QA) on SciQ (val)
62.6AUROCInternal Confidence
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Internal ConfidenceBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 62.6 | 23.6 | 18.2 | |
| P(YES) (top right)Backbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 61.3 | 24.4 | 5.9 | |
| Internal ConfidenceBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 60.7 | 25.8 | 10.4 | |
| PerplexityBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 60.1 | 20.7 | — | |
| P(YES) (top right)Backbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 60 | 21.7 | 24.1 | |
| P(YES) (naive avg)Backbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 59.5 | 17.9 | 14.6 | |
| Attentional EntropyBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 59.4 | 19.2 | — | |
| P(YES) (top right)Backbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 58.4 | 17.2 | 23.7 | |
| PerplexityBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 58.3 | 15.1 | — | |
| Internal ConfidenceBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 58.1 | 15.7 | 16.7 | |
| P(YES) (naive avg)Backbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 57.9 | 14.9 | 20.4 | |
| PerplexityBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 57.7 | 16.6 | — | |
| Attentional EntropyBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 57.7 | 15.2 | — | |
| P(YES) (naive avg)Backbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 57.5 | 18.8 | 6.4 | |
| Attentional EntropyBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 56.2 | 9.4 | — | |
| Min-K EntropyBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 55.7 | 11.3 | — | |
| Max(− log p)Backbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 54.1 | 6.9 | — | |
| Min-K EntropyBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 53.5 | 7.9 | — | |
| Predictive EntropyBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 53.2 | 6.9 | — | |
| Min-K EntropyBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 52.7 | 4.9 | — | |
| Max(− log p)Backbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 51.4 | 2.9 | — | |
| Max(− log p)Backbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 51.4 | 1.9 | — | |
| Predictive EntropyBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 51.4 | 3.2 | — | |
| Predictive EntropyBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 51.2 | 3.9 | — | |
| Internal Semantic SimilarityBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 46.9 | -5.9 | 12.2 | |
| Internal Semantic SimilarityBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 46.1 | -7.1 | 30.8 | |
| Internal Semantic SimilarityBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 45.5 | -7.7 | 14.9 |