Reading Comprehension on DROP (test)
96.42F1 ScoreHuman Performance
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Human Performance2019.10 | 96.42 | 94.09 | |
| Human2020.09 | 96.42 | 94.09 | |
| Human2019.03 | 96.42 | 94.09 | |
| QDGATbenchmark-specific tuning=true2023.03 | 88.4 | — | |
| QDGATpEnsemble=true, Backbone=RoBERTa, Pre-training=SQuAD2020.09 | 88.38 | 85.46 | |
| QDGATpBackbone=RoBERTa, Pre-training=SQuAD2020.09 | 87.57 | 84.53 | |
| QDGATBackbone=RoBERTa2020.09 | 86.38 | 83.23 | |
| NumNet+Ensemble=true2020.09 | 86.16 | 83.14 | |
| NumNet+Backbone=RoBERTa2020.09 | 84.84 | 81.52 | |
| MANGOBase LLM=GPT-4o-mini2026.05 | 84.75 | — | |
| ALBERT-CalculatorBackbone=ALBERT2020.09 | 83.56 | 79.85 | |
| MANGO (Skip-2)Base LLM=GPT-4o-mini2026.05 | 82.91 | — | |
| GPT-4number_of_shots=3-shot2023.03 | 80.9 | — | |
| AFlowBase LLM=GPT-4o-mini2026.05 | 80.63 | — | |
| DyLANBase LLM=GPT-4o-mini2026.05 | 80.29 | — | |
| MaASBase LLM=GPT-4o-mini2026.05 | 80.25 | — | |
| Plan-and-ExecuteBase LLM=GPT-4o-mini2026.05 | 80.13 | — | |
| GPTSwarmBase LLM=GPT-4o-mini2026.05 | 79.77 | — | |
| SC (CoT×5)Base LLM=GPT-4o-mini2026.05 | 78.73 | — | |
| CoTBase LLM=GPT-4o-mini2026.05 | 78.5 | — | |
| AgentSquareBase LLM=GPT-4o-mini2026.05 | 78.38 | — | |
| ADASBase LLM=GPT-4o-mini2026.05 | 76.63 | — | |
| MultiPersonaBase LLM=GPT-4o-mini2026.05 | 74.38 | — | |
| PaLMnumber_of_shots=1-shot2023.03 | 70.8 | — | |
| Self-RefineBase LLM=GPT-4o-mini2026.05 | 70.27 | — | |
| IO (direct LLM invocation)Base LLM=GPT-4o-mini2026.05 | 68.25 | — | |
| NumNetApproach=Numerical MRC2019.10 | 67.97 | 64.56 | |
| NumNet2020.09 | 67.97 | 64.56 | |
| NAQANet+Approach=Numerical MRC2019.10 | 64.29 | 60.82 | |
| GPT-3.5number_of_shots=3-shot2023.03 | 64.1 | — | |
| LLM-DebateBase LLM=GPT-4o-mini2026.05 | 60.63 | — | |
| QuanTABase model=LLaMA2 7B, # Params (%)=0.261%, axis_dimensions=16-16-162024.05 | 59.6 | — | |
| QuanTABase model=LLaMA2 7B, # Params (%)=0.041%, axis_dimensions=16-8-8-42024.05 | 59.5 | — | |
| FTBase model=LLaMA2 7B, # Params (%)=100%2024.05 | 59.4 | — | |
| LoRETTABase model=LLaMA2 7B, # Params (%)=1.254%, rank=1282024.05 | 59.1 | — | |
| ParallelBase model=LLaMA2 7B, # Params (%)=0.747%2024.05 | 59 | — | |
| MoRABase model=LLaMA2 7B, # Params (%)=0.996%, rank=1282024.05 | 58.9 | — | |
| SeriesBase model=LLaMA2 7B, # Params (%)=0.747%2024.05 | 58.8 | — | |
| MoRABase model=LLaMA2 7B, # Params (%)=0.062%, rank=82024.05 | 58.6 | — | |
| KronABase model=LLaMA2 7B, # Params (%)=0.996%, matrix_sizes=1024-42024.05 | 58.5 | — | |
| MoRABase model=LLaMA2 7B, # Params (%)=0.249%, rank=322024.05 | 58.2 | — | |
| KronABase model=LLaMA2 7B, # Params (%)=0.062%, matrix_sizes=256-162024.05 | 57.7 | — | |
| LoRABase model=LLaMA2 7B, # Params (%)=0.996%, rank=1282024.05 | 56.2 | — | |
| LoRETTABase model=LLaMA2 7B, # Params (%)=0.083%, rank=322024.05 | 54.9 | — | |
| LoRABase model=LLaMA2 7B, # Params (%)=0.249%, rank=322024.05 | 54.8 | — | |
| LoRABase model=LLaMA2 7B, # Params (%)=0.062%, rank=82024.05 | 54 | — | |
| KronABase model=LLaMA2 7B, # Params (%)=0.008%, matrix_sizes=64-642024.05 | 50.9 | — | |
| LoRETTABase model=LLaMA2 7B, # Params (%)=0.009%, rank=82024.05 | 48.6 | — | |
| NAQANetApproach=Numerical MRC2019.10 | 47.01 | 44.07 | |
| NAQANet2020.09 | 47.01 | 44.07 | |
| NAQANetConfiguration=Complete Model2019.03 | 47.01 | 44.07 | |
| NAQANetConfiguration=+ Add/Sub2019.03 | 42.96 | 40.4 | |
| NAQANetConfiguration=+ Count2019.03 | 32.75 | 30.04 | |
| BERTApproach=Traditional MRC2019.10 | 32.7 | 29.45 | |
| BERTBackbone=BERT2020.09 | 32.7 | 29.45 | |
| BERTCategory=SQuAD-style RC2019.03 | 32.7 | 29.45 | |
| QANet+ELMoCategory=SQuAD-style RC2019.03 | 29.67 | 27.08 | |
| QANetApproach=Traditional MRC2019.10 | 28.36 | 25.5 | |
| QANet2020.09 | 28.36 | 25.5 | |
| QANetCategory=SQuAD-style RC2019.03 | 28.36 | 25.5 | |
| NAQANetConfiguration=+ Q Span2019.03 | 28.18 | 24.98 | |
| BiDAFApproach=Traditional MRC2019.10 | 27.49 | 24.75 | |
| BiDAF2020.09 | 27.49 | 24.75 | |
| BiDAFCategory=SQuAD-style RC2019.03 | 27.49 | 24.75 | |
| SRLApproach=Semantic Parsing2019.10 | 11.45 | 8.98 | |
| SRL2020.09 | 11.45 | 8.98 | |
| SRLCategory=Semantic Parsing2019.03 | 11.45 | 8.98 | |
| Syn DepApproach=Semantic Parsing2019.10 | 10.84 | 8.51 | |
| Syn Dep2020.09 | 10.84 | 8.51 | |
| Syn DepCategory=Semantic Parsing2019.03 | 10.84 | 8.51 | |
| OpenIEApproach=Semantic Parsing2019.10 | 10.77 | 8.53 | |
| OpenIE2020.09 | 10.77 | 8.53 | |
| OpenIECategory=Semantic Parsing2019.03 | 10.77 | 8.53 | |
| Q-onlyCategory=Heuristic Baselines2019.03 | 8.59 | 4.18 | |
| P-onlyCategory=Heuristic Baselines2019.03 | 2.26 | 0.14 | |
| MajorityCategory=Heuristic Baselines2019.03 | 1.44 | 0.07 | |
| T2MBackbone=LLaDA2.1-mini (16B MoE), Decoding Strategy=greedy decoding, Temperature=0, block length=32, LOWPROB parameters=τ=0.3, Cmax=1, ρmax=0.252026.04 | — | 54.41 | |
| T2TBackbone=LLaDA2.1-mini (16B MoE), Decoding Strategy=greedy decoding, Temperature=0, block length=322026.04 | — | 53.98 |