Question Answering on RelExt MRQA out-of-domain evaluation
79.2EMRGX
Evaluation results
| Method | Links | ||
|---|---|---|---|
| RGXSource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 79.2 | 88.6 | |
| ELECTRASource Domain=SQuAD_wiki, Method=Extraction2021.03 | 79 | 88.4 | |
| SynQASource Domain=SQuAD_wiki, Method=Extraction2021.03 | 78.9 | 88.6 | |
| RGX w/o MMISource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=false, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 78.9 | 88.5 | |
| Prompt tuningBackbone=Llama-3.2-3B-Base, Context=with context2026.02 | 77.37 | 86.7 | |
| RGXSource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 76.1 | 87.2 | |
| T5 + RGXSource Domain=SQuAD_wiki, Method=Prompt Tuning + Seq2seq Generation2021.03 | 75.9 | 87.1 | |
| RGX w/o EMSource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=false, Cooperative self-training (CST)=true2021.03 | 75.6 | 85.9 | |
| RGX w/o MMISource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=false, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 75.4 | 86.7 | |
| T5Source Domain=SQuAD_wiki, Method=Prompt Tuning + Seq2seq Generation2021.03 | 74.5 | 86.5 | |
| RGX w/o CSTSource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=false2021.03 | 74.3 | 85.3 | |
| RGX w/o EMSource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=false, Cooperative self-training (CST)=true2021.03 | 74.1 | 86.2 | |
| QAGen2SSource Domain=SQuAD_wiki, Method=Extraction2021.03 | 73.4 | 84.8 | |
| RGX w/o CSTSource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=false2021.03 | 72.7 | 85.4 | |
| Full-FTEvaluation Protocol=Fine-tuning2025.10 | 72.46 | 83.34 | |
| QAGen2SSource Domain=NaturalQuestions_wiki, Method=Extraction2021.03 | 71.6 | 84.4 | |
| ICAEBackbone=Llama-3.2-3B-Base2026.02 | 70.66 | 82.1 | |
| Prompt tuningBackbone=Llama-3.2-1B-Base, Context=with context2026.02 | 69.91 | 81.15 | |
| ELECTRASource Domain=NaturalQuestions_wiki, Method=Extraction2021.03 | 67.7 | 81.8 | |
| ComprExITBackbone=Llama-3.2-3B-Base2026.02 | 65.31 | 77.25 | |
| 500xBackbone=Llama-3.2-3B-Base2026.02 | 65.13 | 79.67 | |
| ICAEBackbone=Llama-3.2-1B-Base2026.02 | 62.28 | 75.97 | |
| ComprExITBackbone=Llama-3.2-1B-Base2026.02 | 59.46 | 74.96 | |
| BeaconBackbone=Llama-3.2-3B-Base2026.02 | 58.92 | 71.62 | |
| Zero-shotBackbone=Llama-3.2-3B-Base, Context=with context2026.02 | 54.24 | 68.46 | |
| SACCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 47.9 | 61.04 | |
| EPLCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 46.34 | 59.75 | |
| 500xCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 41.11 | 54.37 | |
| ICAECompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 40.33 | 55.24 | |
| DASTCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 35.92 | 48.54 | |
| BeaconBackbone=Llama-3.2-1B-Base2026.02 | 33.18 | 48.82 | |
| Zero-shotBackbone=Llama-3.2-1B-Base, Context=with context2026.02 | 27.48 | 47.87 | |
| BeaconCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 23.91 | 53.28 | |
| Lingua-2Compression Ratio=5x, Evaluation Protocol=Fine-tuning2025.10 | 20.59 | 39.3 | |
| Zero-shotBackbone=Llama-3.2-3B-Base, Context=without context2026.02 | 14.28 | 21.03 | |
| 500xBackbone=Llama-3.2-1B-Base2026.02 | 6.31 | 16.3 | |
| Zero-shotBackbone=Llama-3.2-1B-Base, Context=without context2026.02 | 5.09 | 10.33 |