Reliability prediction on POPE Adversarial
97.1AUROCHidden-state probe (best layer)
Evaluation results
| Method | Links | |
|---|---|---|
| Hidden-state probe (best layer)Model=Qwen2-VL, Method Category=Internal readouts (single pass)2026.05 | 97.1 | |
| Stacked probe (last 5 layers)Model=Qwen2-VL, Method Category=Internal readouts (single pass)2026.05 | 97 | |
| Stacked probe (last 5 layers)Model=LLaVA-1.5, Method Category=Internal readouts (single pass)2026.05 | 95.7 | |
| Hidden-state probe (best layer)Model=LLaVA-1.5, Method Category=Internal readouts (single pass)2026.05 | 95.6 | |
| Self-consistency (K=10)Model=PaliGemma, Method Category=Behavioral (10× cost), K=102026.05 | 80.8 | |
| Self-consistency (K=10)Model=Qwen2-VL, Method Category=Behavioral (10× cost), K=102026.05 | 79.3 | |
| Self-consistency (K=10)Model=LLaVA-1.5, Method Category=Behavioral (10× cost), K=102026.05 | 78.2 | |
| Stacked probe (last 5 layers)Model=PaliGemma, Method Category=Internal readouts (single pass)2026.05 | 75.2 | |
| Hidden-state probe (best layer)Model=PaliGemma, Method Category=Internal readouts (single pass)2026.05 | 73.8 | |
| Truth margin ΔMℓ aloneModel=LLaVA-1.5, Method Category=Internal readouts (single pass)2026.05 | 72.2 | |
| Truth margin ΔMℓ aloneModel=PaliGemma, Method Category=Internal readouts (single pass)2026.05 | 70.1 | |
| Truth margin ΔMℓ aloneModel=Qwen2-VL, Method Category=Internal readouts (single pass)2026.05 | 63.4 | |
| Output confidenceModel=PaliGemma, Method Category=Single-pass baselines2026.05 | 55.4 | |
| Output confidenceModel=LLaVA-1.5, Method Category=Single-pass baselines2026.05 | 54.1 | |
| Output confidenceModel=Qwen2-VL, Method Category=Single-pass baselines2026.05 | 53.2 | |
| Logit entropyModel=PaliGemma, Method Category=Single-pass baselines2026.05 | 52.1 | |
| Logit entropyModel=Qwen2-VL, Method Category=Single-pass baselines2026.05 | 50.8 | |
| Spatial attention (Hs, Ck)Model=PaliGemma, Method Category=Single-pass baselines2026.05 | 50.5 | |
| Spatial attention (Hs, Ck)Model=LLaVA-1.5, Method Category=Single-pass baselines2026.05 | 50.3 | |
| Logit entropyModel=LLaVA-1.5, Method Category=Single-pass baselines2026.05 | 50.2 | |
| Spatial attention (Hs, Ck)Model=Qwen2-VL, Method Category=Single-pass baselines2026.05 | 50.1 |