Question Answering on TextbookQA MRQA (out-of-domain evaluation)
60.48EMPrompt tuning
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Prompt tuningBackbone=Llama-3.2-3B-Base, Context=with context2026.02 | 60.48 | 68.75 | |
| ComprExITBackbone=Llama-3.2-3B-Base2026.02 | 56.09 | 64.5 | |
| Prompt tuningBackbone=Llama-3.2-1B-Base, Context=with context2026.02 | 52.1 | 59.61 | |
| RGXSource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 51.2 | 61.2 | |
| RGX w/o EMSource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=false, Cooperative self-training (CST)=true2021.03 | 50.6 | 58.9 | |
| RGX w/o MMISource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=false, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 50.1 | 60.4 | |
| RGXSource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 49.9 | 60.9 | |
| RGX w/o MMISource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=false, EM selection (difficult questions)=true, Cooperative self-training (CST)=true2021.03 | 49.4 | 60.6 | |
| RGX w/o CSTSource Domain=SQuAD_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=false2021.03 | 48.6 | 57 | |
| QAGen2SSource Domain=SQuAD_wiki, Method=Extraction2021.03 | 48 | 56.5 | |
| RGX w/o EMSource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=false, Cooperative self-training (CST)=true2021.03 | 47.4 | 59.8 | |
| ELECTRASource Domain=SQuAD_wiki, Method=Extraction2021.03 | 43 | 53.6 | |
| ICAEBackbone=Llama-3.2-3B-Base2026.02 | 42.25 | 50.7 | |
| RGX w/o CSTSource Domain=NaturalQuestions_wiki, Method=Extraction, Maximum Mutual Information (MMI)=true, EM selection (difficult questions)=true, Cooperative self-training (CST)=false2021.03 | 41.9 | 53.8 | |
| ComprExITBackbone=Llama-3.2-1B-Base2026.02 | 41.65 | 50.73 | |
| SynQASource Domain=SQuAD_wiki, Method=Extraction2021.03 | 41.4 | 50.2 | |
| T5 + RGXSource Domain=SQuAD_wiki, Method=Prompt Tuning + Seq2seq Generation2021.03 | 41.1 | 64.2 | |
| QAGen2SSource Domain=NaturalQuestions_wiki, Method=Extraction2021.03 | 39.9 | 51.7 | |
| T5Source Domain=SQuAD_wiki, Method=Prompt Tuning + Seq2seq Generation2021.03 | 37.9 | 61.9 | |
| Zero-shotBackbone=Llama-3.2-3B-Base, Context=with context2026.02 | 36.46 | 46.66 | |
| Full-FTEvaluation Protocol=Fine-tuning2025.10 | 32.4 | 53.32 | |
| ELECTRASource Domain=NaturalQuestions_wiki, Method=Extraction2021.03 | 31.9 | 41.5 | |
| 500xBackbone=Llama-3.2-3B-Base2026.02 | 29.41 | 38.1 | |
| ICAEBackbone=Llama-3.2-1B-Base2026.02 | 28.21 | 34.6 | |
| SACCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 28.21 | 44.21 | |
| 500xCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 25.82 | 41.09 | |
| EPLCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 25.42 | 41.31 | |
| Zero-shotBackbone=Llama-3.2-3B-Base, Context=without context2026.02 | 23.29 | 28.89 | |
| DASTCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 22.75 | 36.69 | |
| ICAECompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 21.56 | 34.75 | |
| Zero-shotBackbone=Llama-3.2-1B-Base, Context=with context2026.02 | 18.16 | 29.97 | |
| 500xBackbone=Llama-3.2-1B-Base2026.02 | 16.77 | 24.4 | |
| Lingua-2Compression Ratio=5x, Evaluation Protocol=Fine-tuning2025.10 | 15.83 | 28.42 | |
| BeaconCompression Ratio=15x, Evaluation Protocol=Fine-tuning2025.10 | 12.97 | 37.27 | |
| Zero-shotBackbone=Llama-3.2-1B-Base, Context=without context2026.02 | 11.11 | 16.03 | |
| BeaconBackbone=Llama-3.2-3B-Base2026.02 | 10.98 | 15.68 | |
| BeaconBackbone=Llama-3.2-1B-Base2026.02 | 5.92 | 10.34 |