Long Form Question Answering on ELI5 (test)
27.13ROUGE-LRBG
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| RBG2022.03 | 27.13 | — | — | 24.53 | 2.62 | 10.83 | 27.25 | — | — | |
| RBG2022.03 | 26.46 | — | — | 29.04 | — | — | — | — | — | |
| FiDBackbone=BART2022.03 | 25.7 | — | — | 28.55 | — | — | — | — | — | |
| BART2019.10 | 24.3 | 30.6 | 6.2 | — | — | — | — | — | — | |
| Multi-task Graph + Top-100 Attention + E-MCAInput Length=11K2019.10 | 24 | 30 | 5.8 | — | — | — | — | — | — | |
| Multi-task Graph + Top-100 AttentionInput Length=8502019.10 | 23.8 | 29.7 | 5.7 | — | — | — | — | — | — | |
| Multi-task GraphInput Length=8502019.10 | 23.6 | 29.5 | 5.6 | — | — | — | — | — | — | |
| Transformer Multitask + LayerDropEnc=6, Dec=62019.09 | 23.4 | 29.4 | 5.5 | — | — | — | — | — | — | |
| Q + D to A GraphInput Length=8502019.10 | 23.3 | 28.8 | 5.3 | — | — | — | — | — | — | |
| Multi-task TriplesInput Length=8502019.10 | 23.2 | 29 | 5.2 | — | — | — | — | — | — | |
| Multi-task Top20 Trip.Input Length=avg 5702019.10 | 23.2 | 28.8 | 5.3 | — | — | — | — | — | — | |
| RT + C-REALM2021.03 | 23.2 | — | — | 22.9 | — | — | — | — | — | |
| Transformer MultitaskEnc=6, Dec=62019.09 | 23.1 | 28.9 | 5.4 | — | — | — | — | — | — | |
| Language Model2019.10 | 23.1 | 27.8 | 4.7 | — | — | — | — | — | — | |
| Seq2Seq Multitask2019.10 | 23.1 | 28.9 | 5.4 | — | — | — | — | — | — | |
| Multi-taskInput Length=avg 8502019.10 | 23.1 | 28.9 | 5.4 | — | — | — | — | — | — | |
| Q + D to A, MMRInput Length=avg 8502019.10 | 22.9 | 28.1 | 5 | — | — | — | — | — | — | |
| c-REALM2022.03 | 22.88 | — | — | 23.19 | 2.36 | 10.67 | 24.56 | — | — | |
| Seq2Seq2019.10 | 22.8 | 28.3 | 5.1 | — | — | — | — | — | — | |
| Q + D to A, TF-IDFInput Length=avg 8502019.10 | 22.8 | 28.3 | 5.1 | — | — | — | — | — | — | |
| BARTModel Size=large2022.03 | 22.69 | — | — | 22.19 | — | — | — | — | — | |
| T5Model Size=base2022.03 | 21.02 | — | — | 18.36 | — | — | — | — | — | |
| EMAT-SKSVModel Category=Ours, Q/s=712022.10 | 20.91 | — | — | 19.03 | — | — | — | — | — | |
| EMAT-FKSVModel Category=Ours, Q/s=672022.10 | 20.61 | — | — | 18.42 | — | — | — | — | — | |
| BART-largeModel Category=Parametric models, Q/s=302022.10 | 20.55 | — | — | 19.23 | — | — | — | — | — | |
| BART-large2022.03 | 19.23 | — | — | 20.55 | 0 | 0 | 0 | — | — | |
| T5-baseModel Category=Parametric models, Q/s=762022.10 | 19.08 | — | — | 16.01 | — | — | — | — | — | |
| longest top-1 train answer2021.03 | 18.7 | — | — | 21.6 | — | — | — | — | — | |
| longest top-7 train answer2021.03 | 18.5 | — | — | 22 | — | — | — | — | — | |
| DPR+BART2022.03 | 17.88 | — | — | 17.41 | 1.9 | 10.67 | 26.92 | — | — | |
| Best Extractive2019.10 | 17.5 | 23.5 | 3.1 | — | — | — | — | — | — | |
| DPR+BART2022.03 | 17.41 | — | — | 17.88 | — | — | — | — | — | |
| BART + DPRModel Category=Retrieval-augmented models, Q/s=0.22022.10 | 17.41 | — | — | 17.88 | — | — | — | — | — | |
| BART + DPRYear=20202021.03 | 17.4 | — | — | 17.9 | — | — | — | — | — | |
| copy inputBound Type=Lower bound2021.03 | 16.9 | — | — | 14.8 | — | — | — | — | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-3, Size (MB)=1102022.03 | 16.27 | 26.38 | 5.13 | — | — | — | — | — | — | |
| DQ-BART (Distillation-Aware Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-6, Size (MB)=1372022.03 | 16.23 | 27.1 | 5.15 | — | — | — | — | — | — | |
| DQ-BART (Distillation-Aware Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=6-6, Size (MB)=392022.03 | 16.15 | 26.33 | 4.97 | — | — | — | — | — | — | |
| RAG2022.03 | 16.11 | — | — | 17.24 | — | — | — | — | — | |
| T5-base2022.03 | 16.1 | — | — | 19.08 | 0 | 0 | 0 | — | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=6-3, Size (MB)=322022.03 | 15.94 | 25.41 | 4.83 | — | — | — | — | — | — | |
| DPR_kilt_wiki2022.03 | 15.91 | — | — | 16.45 | 2.46 | 14.83 | 27.69 | — | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-1, Size (MB)=922022.03 | 15.71 | 24.27 | 4.74 | — | — | — | — | — | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=3-1, Size (MB)=722022.03 | 15.51 | 23.69 | 4.53 | — | — | — | — | — | — | |
| random train answerBound Type=Lower bound2021.03 | 15.5 | — | — | 17.1 | — | — | — | — | — | |
| BART (Full Precision)W-E-A (#bits)=32-32-32, E-D (#layers)=6-6, Size (MB)=5312022.03 | 15.36 | 26.02 | 5.11 | — | — | — | — | — | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=6-1, Size (MB)=272022.03 | 15.2 | 23.34 | 4.31 | — | — | — | — | — | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=3-1, Size (MB)=222022.03 | 14.95 | 22.6 | 3.99 | — | — | — | — | — | — | |
| RAG2022.03 | 14.51 | — | — | 14.05 | 1.69 | 11 | 22.92 | — | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=1-1, Size (MB)=192022.03 | 14.3 | 21.51 | 3.44 | — | — | — | — | — | — | |
| RAGYear=2020c2021.03 | 14.1 | — | — | 14.5 | — | — | — | — | — | |
| RAGModel Category=Retrieval-augmented models, Q/s=0.42022.10 | 14.05 | — | — | 14.51 | — | — | — | — | — | |
| BART (Direct Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-6, Size (MB)=1372022.03 | 4.89 | 6.72 | 0.43 | — | — | — | — | — | — | |
| RePAQ w/ EMAT key encoderModel Category=Retrieval-only models2022.10 | 1.65 | — | — | 1.4 | — | — | — | — | — | |
| INFO-RAGBackbone=LLaMA-2-7B, Zero-shot=true2024.02 | — | — | — | — | — | — | — | 17.18 | 39.83 | |
| INFO-RAGBackbone=LLaMA-2-7B-chat, Zero-shot=true2024.02 | — | — | — | — | — | — | — | 28.15 | 43.78 | |
| INFO-RAGBackbone=LLaMA-2-13B, Zero-shot=true2024.02 | — | — | — | — | — | — | — | 17.48 | 41.04 | |
| INFO-RAGBackbone=LLaMA-2-13B-chat, Zero-shot=true2024.02 | — | — | — | — | — | — | — | 27.24 | 46.55 | |
| LLaMA-2-13BZero-shot=true2024.02 | — | — | — | — | — | — | — | 14.8 | 36.86 | |
| LLaMA-2-13B-chatZero-shot=true2024.02 | — | — | — | — | — | — | — | 27.07 | 43.23 | |
| LLaMA-2-7BZero-shot=true2024.02 | — | — | — | — | — | — | — | 15.18 | 35.78 | |
| LLaMA-2-7B-chatZero-shot=true2024.02 | — | — | — | — | — | — | — | 27.81 | 40.83 |