Question Answering on SQuAD v1.1
94.7F1Adafactor
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AdafactorPrecision=32-bit, Backbone=RoBERTa-Large, Embedding Layer Quantization=False2023.09 | 94.7 | 89 | |
| AdamWPrecision=32-bit, Backbone=RoBERTa-Large2023.09 | 94.6 | 89 | |
| AdafactorPrecision=32-bit, Backbone=RoBERTa-Large2023.09 | 94.6 | 88.8 | |
| FactorPrecision=4-bit, Backbone=RoBERTa-Large2023.09 | 94.6 | 88.8 | |
| AdamWPrecision=8-bit, Backbone=RoBERTa-Large, Embedding Layer Quantization=False2023.09 | 94.5 | 88.8 | |
| AdamWPrecision=4-bit, Backbone=RoBERTa-Large2023.09 | 94.5 | 88.8 | |
| FLoRGBase Model=Llama-3.2-3B2026.02 | 92.74 | 89.83 | |
| RoBERTa_BASEPre-train (P)=true2022.10 | 92.2 | — | |
| XDocPre-training Steps=1M, Pre-train (P)=true, Pre-train (D)=true, Pre-train (W)=true2022.10 | 92 | — | |
| XDocPre-training Steps=100K, Pre-train (P)=true2022.10 | 91.9 | — | |
| XDocPre-training Steps=100K, Pre-train (P)=true, Pre-train (D)=true2022.10 | 91.9 | — | |
| FedEx-LoRABase Model=Llama-3.2-3B2026.02 | 91.82 | 88.25 | |
| XDocPre-training Steps=100K, Pre-train (D)=true, Pre-train (W)=true2022.10 | 91.7 | — | |
| XDocPre-training Steps=100K, Pre-train (P)=true, Pre-train (D)=true, Pre-train (W)=true2022.10 | 91.7 | — | |
| XDocPre-training Steps=500K, Pre-train (P)=true, Pre-train (D)=true, Pre-train (W)=true2022.10 | 91.7 | — | |
| SM3Precision=32-bit, Backbone=RoBERTa-Large2023.09 | 91.7 | 84.2 | |
| XDocPre-training Steps=100K, Pre-train (P)=true, Pre-train (W)=true2022.10 | 91.6 | — | |
| XDocPre-training Steps=100K, Pre-train (D)=true2022.10 | 91.4 | — | |
| XDocPre-training Steps=100K, Pre-train (W)=true2022.10 | 91.4 | — | |
| FedSA-LoRABase Model=Llama-3.2-3B2026.02 | 91.35 | 87.12 | |
| FLoRGBase Model=RoBERTa-large2026.02 | 91.15 | 86.34 | |
| FeDeRABase Model=Llama-3.2-3B2026.02 | 91.08 | 86.58 | |
| FedEx-LoRABase Model=RoBERTa-large2026.02 | 90.84 | 86.02 | |
| FFA-LoRABase Model=Llama-3.2-3B2026.02 | 90.31 | 85.37 | |
| FedSA-LoRABase Model=RoBERTa-large2026.02 | 90.23 | 84.47 | |
| FeDeRABase Model=RoBERTa-large2026.02 | 90.03 | 83.12 | |
| FedITBase Model=Llama-3.2-3B2026.02 | 89.53 | 84.21 | |
| ALBERT-BASE+AA-CTPre-trained Model=ALBERT-BASE, Alignment Attention Method=CT2021.10 | 89.02 | 81.32 | |
| ALBERT-BASE+AA-GANPre-trained Model=ALBERT-BASE, Alignment Attention Method=GAN2021.10 | 88.92 | 81.19 | |
| ALBERT-BASE+AA-OTPre-trained Model=ALBERT-BASE, Alignment Attention Method=Optimal Transport (OT)2021.10 | 88.89 | 81.13 | |
| CAMEmode=fine-tuning, batch size=8k2023.07 | 88.8 | 81.8 | |
| ALBERT-BASEBackbone=ALBERT-base2021.10 | 88.7 | 80.86 | |
| ALBERT-BASEPre-trained Model=ALBERT-BASE, Alignment Attention Method=None (Soft Attention)2021.10 | 88.7 | 80.86 | |
| FFA-LoRABase Model=RoBERTa-large2026.02 | 88.68 | 81.69 | |
| Baselinemode=fine-tuning2023.07 | 88.5 | 80.8 | |
| CAMEmode=fine-tuning, batch size=32k2023.07 | 88.5 | 81.2 | |
| RoCoFT-RowBackbone=DeBERTaV3-base, Trainable Parameters=0.161M2024.10 | 88.15 | 81.7 | |
| FTBackbone=DeBERTaV3-base, Trainable Parameters=184M2024.10 | 88.14 | 82.83 | |
| RoCoFT-ColumnBackbone=DeBERTaV3-base, Trainable Parameters=0.161M2024.10 | 88.11 | 81.63 | |
| FedITBase Model=RoBERTa-large2026.02 | 88.09 | 81.53 | |
| LoHaBackbone=DeBERTaV3-base, Trainable Parameters=0.765M2024.10 | 88.02 | 81.43 | |
| OLTQAConfiguration=Full2023.05 | 87.88 | — | |
| ExactOBSModel=BERT, FLOP reduction target=2x2022.08 | 87.81 | — | |
| AdaLoRABackbone=DeBERTaV3-base, Trainable Parameters=0.810M2024.10 | 87.75 | 81.16 | |
| OLTQAKnowledge Distillation=Static MKD2023.05 | 87.72 | — | |
| OLTQAKnowledge Distillation=without MKD2023.05 | 87.39 | — | |
| LoRABackbone=DeBERTaV3-base, Trainable Parameters=0.740M2024.10 | 87.16 | 81.64 | |
| Hyperformer++2023.05 | 87.13 | — | |
| AdaPruneModel=BERT, FLOP reduction target=2x2022.08 | 87.12 | — | |
| EPR2023.05 | 87.12 | — | |
| OLTQARetriever=EPR2023.05 | 87.09 | — | |
| OLTQARe-ranker=excluded2023.05 | 86.73 | — | |
| OLTQAKnowledge Distillation=Back KD2023.05 | 86.66 | — | |
| LoKrBackbone=DeBERTaV3-base, Trainable Parameters=0.815M2024.10 | 86.45 | 80.64 | |
| BitfitBackbone=DeBERTaV3-base, Trainable Parameters=0.172M2024.10 | 86.25 | 80.53 | |
| OLTQAAblation=without Pk2023.05 | 86.02 | — | |
| OLTQAAblation=without Pm2023.05 | 85.98 | — | |
| ExactOBSModel=BERT, FLOP reduction target=3x2022.08 | 85.87 | — | |
| Muppet2023.05 | 85.64 | — | |
| UnifiedQA2023.05 | 85.32 | — | |
| OLTQARetriever=BM252023.05 | 84.96 | — | |
| ProQA2023.05 | 84.33 | — | |
| AS-LoRADP Budget (ϵ)=32026.05 | 83.99 | 73.98 | |
| AS-LoRADP Budget (ϵ)=12026.05 | 83.23 | 72.78 | |
| Prefix-tuningBackbone=DeBERTaV3-base, Trainable Parameters=1.733M2024.10 | 82.94 | 78.38 | |
| FFA-LoRADP Budget (ϵ)=32026.05 | 82.83 | 72.3 | |
| ExactOBSModel=BERT, FLOP reduction target=4x2022.08 | 82.1 | — | |
| FFA-LoRADP Budget (ϵ)=12026.05 | 81.24 | 70.42 | |
| RoLoRADP Budget (ϵ)=32026.05 | 80.63 | 69.75 | |
| Prompt tuningBackbone=DeBERTaV3-base, Trainable Parameters=0.650M2024.10 | 78.42 | 74.52 | |
| FLoRGBase Model=OPT-125M2026.02 | 78.23 | 68.52 | |
| L-OBSModel=BERT, FLOP reduction target=2x2022.08 | 77.67 | — | |
| FedSA-LoRABase Model=OPT-125M2026.02 | 77.31 | 67.14 | |
| FedEx-LoRABase Model=OPT-125M2026.02 | 76.18 | 66.85 | |
| FeDeRABase Model=OPT-125M2026.02 | 74.52 | 64.78 | |
| RoLoRADP Budget (ϵ)=12026.05 | 73.94 | 61.69 | |
| FFA-LoRABase Model=OPT-125M2026.02 | 73.04 | 63.19 | |
| FedITBase Model=OPT-125M2026.02 | 72.08 | 62.31 | |
| AdaPruneModel=BERT, FLOP reduction target=3x2022.08 | 70.32 | — | |
| GMPModel=BERT, FLOP reduction target=2x2022.08 | 65.64 | — | |
| AdaPruneModel=BERT, FLOP reduction target=4x2022.08 | 18.75 | — | |
| GMPModel=BERT, FLOP reduction target=3x2022.08 | 12.52 | — | |
| GMPModel=BERT, FLOP reduction target=4x2022.08 | 9.23 | — | |
| L-OBSModel=BERT, FLOP reduction target=4x2022.08 | 6.63 | — | |
| L-OBSModel=BERT, FLOP reduction target=3x2022.08 | 3.62 | — |