Uncertainty Estimation (Factual QA) on TriviaQA 1,000 samples (val)
71.9AUROCInternal Confidence
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Internal ConfidenceBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 71.9 | 43.3 | 26.5 | |
| Internal ConfidenceBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 68.7 | 35.5 | 25.4 | |
| P(YES) (top right)Backbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 67.8 | 36 | 30.3 | |
| P(YES) (naive avg)Backbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 67 | 33.9 | 3.5 | |
| P(YES) (naive avg)Backbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 65.9 | 33 | 12.6 | |
| P(YES) (top right)Backbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 64.9 | 27.7 | 5.4 | |
| Internal ConfidenceBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 64.7 | 30.1 | 7.9 | |
| P(YES) (naive avg)Backbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 64.1 | 28.3 | 17 | |
| PerplexityBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 61.8 | 24.3 | — | |
| Attentional EntropyBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 60.6 | 21.4 | — | |
| Min-K EntropyBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 59.9 | 20 | — | |
| Min-K EntropyBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 59.9 | 20 | — | |
| Attentional EntropyBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 59.4 | 18.7 | — | |
| Predictive EntropyBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 59.3 | 18.9 | — | |
| Attentional EntropyBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 59.1 | 17.2 | — | |
| PerplexityBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 59.1 | 17.8 | — | |
| Predictive EntropyBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 58.9 | 17.9 | — | |
| PerplexityBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 58.6 | 17.1 | — | |
| Predictive EntropyBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 58.5 | 17.7 | — | |
| Min-K EntropyBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 58.1 | 17.4 | — | |
| Max(− log p)Backbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 56.5 | 12.4 | — | |
| Max(− log p)Backbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 55.5 | 10 | — | |
| P(YES) (top right)Backbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 55.4 | 10.2 | 31.7 | |
| Max(− log p)Backbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 54.9 | 11.1 | — | |
| Internal Semantic SimilarityBackbone=Qwen-14B, Evaluation Shots=0-shot2025.06 | 51 | 2.5 | 2 | |
| Internal Semantic SimilarityBackbone=Phi-3.8B, Evaluation Shots=2-shot2025.06 | 48.7 | -2.4 | 0.3 | |
| Internal Semantic SimilarityBackbone=Llama-8B, Evaluation Shots=0-shot2025.06 | 44.1 | -14.4 | 24.4 |