Question Answering on NQ, TriviaQA, and WebQ (test)
65.5AccuracyChatQA + DTA
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ChatQA + DTABackbone=Llama-3-8b, Abstention Strategy=Divide-Then-Align2025.05 | 65.5 | 64.5 | 67.2 | 65.8 | 62.8 | 48.9 | 67.9 | 61.8 | 64.7 | |
| Ret-Robust + DTABackbone=Llama-2-13b, Abstention Strategy=Divide-Then-Align2025.05 | 64.8 | 67.9 | 65.3 | 66.6 | 76.8 | 45.5 | 56.7 | 63.5 | 59.9 | |
| RAAT + DTABackbone=Llama-2-7b, Abstention Strategy=Divide-Then-Align2025.05 | 64.1 | 63.7 | 65.5 | 64.6 | 68.9 | 52.8 | 65 | 61.7 | 63.3 | |
| Ret-Robust + ConsistencyBackbone=Llama-2-13b, Abstention Strategy=Consistency2025.05 | 53.9 | 71.8 | 54 | 61.7 | 89.6 | 46.4 | 6.3 | 52.5 | 11.2 | |
| Ret-Robust + LogitsBackbone=Llama-2-13b, Abstention Strategy=Logits2025.05 | 53.6 | 70 | 53.6 | 60.7 | 87.9 | 43.4 | 10 | 52.9 | 16.9 | |
| Ret-Robust + ICLBackbone=Llama-2-13b, Abstention Strategy=ICL2025.05 | 52 | 71.6 | 52 | 60.3 | 89.1 | 46.6 | 0 | 0 | 0 | |
| Ret-RobustBackbone=Llama-2-13b2025.05 | 51.6 | 71 | 51.6 | 59.8 | 90 | 44.5 | 0 | 0 | 0 | |
| RAAT + ConsistencyBackbone=Llama-2-7b, Abstention Strategy=Consistency2025.05 | 51.4 | 69 | 50.7 | 58.5 | 82.1 | 58.8 | 16.3 | 58.4 | 25.4 | |
| Ret-Robust + P(true)Backbone=Llama-2-13b, Abstention Strategy=P(true)2025.05 | 50.9 | 56 | 58.5 | 57.2 | 74.8 | 29.7 | 37.5 | 33.6 | 35.4 | |
| ChatQA + P(true)Backbone=Llama-3-8b, Abstention Strategy=P(true)2025.05 | 50.1 | 45.2 | 55.6 | 49.9 | 46.2 | 29.1 | 61.9 | 42.6 | 50.5 | |
| RAAT + LogitsBackbone=Llama-2-7b, Abstention Strategy=Logits2025.05 | 49.2 | 58.8 | 50.5 | 54.3 | 69.8 | 47 | 30.9 | 45.1 | 36.6 | |
| Llama-2-13b OriginalBackbone=Llama-2-13b2025.05 | 48.1 | 66.3 | 48.1 | 55.8 | 82.1 | 40.7 | 0 | 0 | 0 | |
| RAAT + ICLBackbone=Llama-2-7b, Abstention Strategy=ICL2025.05 | 46.8 | 71.2 | 46.8 | 56.5 | 84.4 | 60.2 | 0 | 0 | 0 | |
| ChatQA + LogitsBackbone=Llama-3-8b, Abstention Strategy=Logits2025.05 | 46.6 | 57.8 | 46.8 | 51.7 | 51 | 44.8 | 19.3 | 44.9 | 27 | |
| ChatQA + ConsistencyBackbone=Llama-3-8b, Abstention Strategy=Consistency2025.05 | 46.5 | 61 | 46.7 | 52.9 | 58.7 | 46.6 | 11.3 | 44 | 18 | |
| RAATBackbone=Llama-2-7b2025.05 | 46.2 | 70.2 | 46.2 | 55.7 | 76.3 | 61.7 | 0 | 0 | 0 | |
| ChatQABackbone=Llama-3-8b2025.05 | 46.1 | 60.9 | 45 | 51.8 | 54.5 | 46.8 | 10.2 | 71.8 | 17.8 | |
| RAAT + P(true)Backbone=Llama-2-7b, Abstention Strategy=P(true)2025.05 | 45 | 65 | 46 | 53.8 | 68.9 | 57.4 | 6.71 | 32.1 | 11 | |
| Llama-3-8b OriginalBackbone=Llama-3-8b2025.05 | 43.9 | 62 | 43.9 | 51.4 | 76 | 42 | 0 | 0 | 0 | |
| ChatQA + ICLBackbone=Llama-3-8b, Abstention Strategy=ICL2025.05 | 43.3 | 55 | 41.4 | 47.2 | 50.3 | 40.7 | 15.1 | 75.4 | 25.1 | |
| Llama-2-7b OriginalBackbone=Llama-2-7b2025.05 | 42.2 | 64.1 | 42.2 | 50.9 | 85.8 | 49.9 | 0 | 0 | 0 |