Question Answering on SQuAD v1.1 & v2.0
90.17SQuAD v1.1 F1Mango
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| MangoTraining Protocol=Training from the Pretrained Model: M(12,384) -> M(12,768), FLOPS (x 1e19)=5.4, Ratio (Saving)=39.2%2023.10 | 90.17 | 78.77 | 84.47 | — | — | |
| LiGOTraining Protocol=Training from the Pretrained Model: M(12,384) -> M(12,768), FLOPS (x 1e19)=5.9, Ratio (Saving)=33.5%2023.10 | 90.09 | 78.34 | 84.22 | — | — | |
| bert2BERTTraining Protocol=Training from the Pretrained Model: M(12,384) -> M(12,768), FLOPS (x 1e19)=5.7, Ratio (Saving)=35.6%2023.10 | 90.02 | 78.99 | 84.5 | — | — | |
| StackBERTTraining Protocol=Training from Scratch, FLOPS (x 1e19)=6.3, Ratio (Saving)=29.5%2023.10 | 89.82 | 78.21 | 84.01 | — | — | |
| ScratchTraining Protocol=Training from Scratch, FLOPS (x 1e19)=8.92023.10 | 89.21 | 77.9 | 83.56 | — | — | |
| LiGOModel=RoBERTa-B, FLOPs Savings=22.1%, Walltime Savings=20.3%2026.03 | 84.7 | 42.3 | 59.9 | 75.5 | 37.3 | |
| BoTModel=RoBERTa-B, FLOPs Savings=54.7%, Walltime Savings=51.5%2026.03 | 84.7 | 43.8 | 60.5 | 74.2 | 39.2 | |
| MangoModel=RoBERTa-B, FLOPs Savings=22.2%, Walltime Savings=20.4%2026.03 | 84.6 | 42.2 | 60.4 | 76.3 | 38.4 | |
| ScratchModel=RoBERTa-B, FLOPs Savings=0.0%, Walltime Savings=0.0%2026.03 | 84.4 | 43.6 | 60.4 | 74.7 | 38.8 | |
| bert2BERTModel=RoBERTa-B, FLOPs Savings=7.2%, Walltime Savings=6.6%2026.03 | 83.8 | 41.5 | 59 | 74.2 | 36.6 | |
| DWTModel=RoBERTa-B2026.03 | 79.18 | 39.72 | 55.75 | 69.87 | 34.22 | |
| MangoModel=RoBERTa-B2026.03 | 78.63 | 38.9 | 54.95 | 68.75 | 33.53 | |
| LiGOModel=RoBERTa-B2026.03 | 78.41 | 38.84 | 54.79 | 68.34 | 33.58 | |
| BoTModel=RoBERTa-S, FLOPs Savings=35.7%, Walltime Savings=35.8%2026.03 | 76.7 | 37.9 | 53.3 | 66.3 | 32.4 | |
| ScratchModel=RoBERTa-S, FLOPs Savings=0.0%, Walltime Savings=0.0%2026.03 | 76.2 | 37.5 | 52.9 | 65.8 | 32 | |
| WSModel=RoBERTa-S, FLOPs Savings=32.4%, Walltime Savings=32.3%2026.03 | 75.3 | 37.6 | 52.4 | 64.7 | 32.1 | |
| DWTModel=RoBERTa-S2026.03 | 54.22 | 26.87 | 35.98 | 42.58 | 20.26 | |
| WSModel=RoBERTa-S2026.03 | 32.72 | 18.04 | 21.83 | 23.75 | 12.79 | |
| bert2BERTModel=RoBERTa-B2026.03 | 24.08 | 10.62 | 13.43 | 13.64 | 5.39 | |
| ScratchModel=RoBERTa-B2026.03 | 18.76 | 8.86 | 10.09 | 8.8 | 3.92 | |
| ScratchModel=RoBERTa-S2026.03 | 18.02 | 8.16 | 9.39 | 7.99 | 3.39 |