Question Answering on ASQA (test)
92.94Correctness EM RecallCLARION
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CLARIONModel=Qwen3-235b-a22b-2502025.09 | 92.94 | — | — | — | — | 50.33 | — | 71.64 | |
| CLARIONModel=DeepSeek-Chat-v3.12025.09 | 90.98 | — | — | — | — | 47.99 | — | 69.48 | |
| CLARIONModel=Gemini-2.5-Flash2025.09 | 89.61 | — | — | — | — | 48.02 | — | 68.81 | |
| CLARION w/o clarification & detectionModel=DeepSeek-Chat-v3.12025.09 | 88.82 | — | — | — | — | 46.23 | — | 67.53 | |
| CLARION w/o clarificationModel=Gemini-2.5-Flash2025.09 | 88.43 | — | — | — | — | 46.09 | — | 67.26 | |
| CLARION w/o clarification & detectionModel=Gemini-2.5-Flash2025.09 | 87.65 | — | — | — | — | 46.04 | — | 66.84 | |
| CLARION w/o clarificationModel=DeepSeek-Chat-v3.12025.09 | 87.45 | — | — | — | — | 46.16 | — | 66.81 | |
| CLARION w/o clarificationModel=Qwen3-235b-a22b-2502025.09 | 85.69 | — | — | — | — | 42.58 | — | 64.13 | |
| CLARION w/o clarification & detectionModel=Qwen3-235b-a22b-2502025.09 | 84.31 | — | — | — | — | 42.3 | — | 63.3 | |
| ReActModel=DeepSeek-Chat-v3.12025.09 | 83.57 | — | — | — | — | 40.01 | — | 61.79 | |
| NaiveRAGModel=Qwen3-235b-a22b-2502025.09 | 83.14 | — | — | — | — | 41.34 | — | 62.24 | |
| ReActModel=Qwen3-235b-a22b-2502025.09 | 82.31 | — | — | — | — | 40.75 | — | 61.53 | |
| ReActModel=Gemini-2.5-Flash2025.09 | 81.13 | — | — | — | — | 41.72 | — | 61.42 | |
| NaiveRAGModel=Gemini-2.5-Flash2025.09 | 76.86 | — | — | — | — | 39.3 | — | 58.08 | |
| DIVAModel=DeepSeek-Chat-v3.12025.09 | 75.1 | — | — | — | — | 25.54 | — | 50.32 | |
| DIVAModel=Qwen3-235b-a22b-2502025.09 | 73.53 | — | — | — | — | 37.54 | — | 55.54 | |
| No RetrievalModel=Qwen3-235b-a22b-2502025.09 | 70.98 | — | — | — | — | 36.56 | — | 53.77 | |
| No RetrievalModel=DeepSeek-Chat-v3.12025.09 | 70.59 | — | — | — | — | 35.57 | — | 53.08 | |
| NaiveRAGModel=DeepSeek-Chat-v3.12025.09 | 67.84 | — | — | — | — | 34.85 | — | 51.34 | |
| No RetrievalModel=Gemini-2.5-Flash2025.09 | 66.86 | — | — | — | — | 34.79 | — | 50.82 | |
| DIVAModel=Gemini-2.5-Flash2025.09 | 66.47 | — | — | — | — | 35.69 | — | 51.08 | |
| FineRefSource Domain=ELI5, Backbone=Mistral-Orca-7B, Evaluation Protocol=Fine-tuned2025.11 | 42.3 | — | 34.7 | 87.2 | 40.5 | 81.3 | — | — | |
| LLAMA-2-7B + f.g.(RS+RL)Backbone=LLAMA-2-7B, Setting=Separate Setting2024.02 | 41.33 | 79.71 | 77.9 | 91.14 | — | — | — | — | |
| LLAMA-2-7B + f.g.RSBackbone=LLAMA-2-7B, Setting=Separate Setting2024.02 | 40.24 | 77.65 | 74.96 | 83.1 | — | — | — | — | |
| +f.g.RSreward_type=fine-grained, optimization=Rejection Sampling2024.02 | 40.07 | — | 74.35 | — | — | — | — | — | |
| LLAMA-2-7B + f.g.RSBackbone=LLAMA-2-7B, Setting=Combined Setting2024.02 | 40.07 | 76.71 | 74.35 | 87.04 | — | — | — | — | |
| +f.g.(RS+RL)reward_type=fine-grained, optimization=RS+RL2024.02 | 40.05 | — | 76.33 | — | — | — | — | — | |
| LLAMA-2-7B + f.g.(RS+RL)Backbone=LLAMA-2-7B, Setting=Combined Setting2024.02 | 40.05 | 77.83 | 76.33 | 89.08 | — | — | — | — | |
| CalfSource Domain=ELI5, Backbone=Mistral-Orca-7B, Evaluation Protocol=Fine-tuned2025.11 | 40 | — | 33.2 | 86.6 | 40.1 | 79.5 | — | — | |
| ICL ChatGPTmode=In-Context Learning (ICL), model=ChatGPT2024.02 | 39.96 | — | 70.97 | — | — | — | — | — | |
| ChatGPTSetting=In-Context Learning2024.02 | 39.96 | 74.72 | 70.97 | 76.49 | — | — | — | — | |
| Zero-ShotBackbone=Mistral-Orca-7B, Evaluation Protocol=Zero-shot2025.11 | 39.5 | — | 31.6 | 78.9 | 39 | 5.7 | — | — | |
| Few-shot FTSource Domain=ELI5, Backbone=Mistral-Orca-7B, Evaluation Protocol=Fine-tuned2025.11 | 38.5 | — | 31.4 | 90.1 | 39.7 | 71.7 | — | — | |
| +h.RSreward_type=heuristic, optimization=Rejection Sampling2024.02 | 37.96 | — | 68.48 | — | — | — | — | — | |
| +h.(RS+RL)reward_type=heuristic, optimization=RS+RL2024.02 | 37.33 | — | 69.37 | — | — | — | — | — | |
| LLAMA-2-7B + f.g.RLBackbone=LLAMA-2-7B, Setting=Separate Setting2024.02 | 36.94 | 71.86 | 69.75 | 77.21 | — | — | — | — | |
| +f.g.RLreward_type=fine-grained, optimization=RL2024.02 | 35.99 | — | 72.38 | — | — | — | — | — | |
| LLAMA-2-7B + f.g.RLBackbone=LLAMA-2-7B, Setting=Combined Setting2024.02 | 35.99 | 76.3 | 72.38 | 69.99 | — | — | — | — | |
| LLAMA-2-13BSetting=In-Context Learning2024.02 | 35.97 | 21.59 | 25.27 | 88.01 | — | — | — | — | |
| Mdistdescription=Base model2024.02 | 35.56 | — | 67.99 | — | — | — | — | — | |
| LLAMA-2-7B MdistBackbone=LLAMA-2-7B, Setting=Combined Setting2024.02 | 35.56 | 74.8 | 67.99 | 62.08 | — | — | — | — | |
| LLAMA-2-7B MdistBackbone=LLAMA-2-7B, Setting=Separate Setting2024.02 | 34.84 | 69.3 | 62.2 | 59.93 | — | — | — | — | |
| +h.RLreward_type=heuristic, optimization=RL2024.02 | 34.33 | — | 70.12 | — | — | — | — | — | |
| LLAMA-2-7BSetting=In-Context Learning2024.02 | 34.15 | 14.12 | 15.26 | 85.31 | — | — | — | — | |
| ICL LLaMA-2-7Bmode=In-Context Learning (ICL), model=LLaMA-2-7B2024.02 | 34.15 | — | 15.26 | — | — | — | — | — | |
| T5 + f.g.RLBackbone=T5, Setting=Separate Setting2024.02 | 33.75 | 74.24 | 68.3 | 61.11 | — | — | — | — | |
| T5 MdistBackbone=T5, Setting=Separate Setting2024.02 | 33.6 | 71.98 | 65.15 | 66.81 | — | — | — | — | |
| Self-RAG 13BSetting=In-Context Learning2024.02 | 31.7 | 71.3 | 70.3 | 71.6 | — | — | — | — | |
| Self-RAG 7BSetting=In-Context Learning2024.02 | 30 | 67.8 | 66.9 | 74.3 | — | — | — | — | |
| Self-RAG 7BBackbone=Mistral-Orca-7B, Evaluation Protocol=Zero-shot2025.11 | 30 | — | — | 74.3 | 35.7 | 67.3 | — | — | |
| ActIdxModel=Q-3B2025.06 | — | 24.2 | 23.9 | — | — | — | 21.4 | — | |
| ActIdxModel=Q-7B2025.06 | — | 31.1 | 30.9 | — | — | — | 27.6 | — | |
| ActIdxModel=Q-14B2025.06 | — | 41.3 | 40.9 | — | — | — | 31.6 | — | |
| ActIdxModel=L-3B2025.06 | — | 29.3 | 29 | — | — | — | 25.9 | — | |
| ActIdxModel=L-8B2025.06 | — | 31.8 | 31.7 | — | — | — | 28.2 | — | |
| InsOnlyModel=Q-3B2025.06 | — | 4.1 | 3.7 | — | — | — | 15.9 | — | |
| InsOnlyModel=Q-7B2025.06 | — | 21.2 | 20 | — | — | — | 19.1 | — | |
| InsOnlyModel=Q-14B2025.06 | — | 27.5 | 25.1 | — | — | — | 26.4 | — | |
| InsOnlyModel=L-3B2025.06 | — | 18.1 | 11.9 | — | — | — | 18 | — | |
| InsOnlyModel=L-8B2025.06 | — | 25.5 | 23.3 | — | — | — | 25.4 | — | |
| PassIdxModel=Q-3B2025.06 | — | 17.4 | 17.1 | — | — | — | 16.5 | — | |
| PassIdxModel=Q-7B2025.06 | — | 24.2 | 24.1 | — | — | — | 21.5 | — | |
| PassIdxModel=Q-14B2025.06 | — | 27.3 | 27.1 | — | — | — | 30 | — | |
| PassIdxModel=L-3B2025.06 | — | 19.1 | 18.5 | — | — | — | 21.5 | — | |
| PassIdxModel=L-8B2025.06 | — | 28.1 | 27.8 | — | — | — | 28.6 | — |