Natural Language Understanding on GLUE (test) (Full Set)
7,564.6QNLIBaseline
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BaselineFramework=BOLT, Fine-tuned=false, Network Setting=MOBILE, Metric Type=Inference Time (s)2026.03 | 7,564.6 | — | — | 7,564.6 | 18,638.96 | 7,564.6 | 7,564.6 | 3,393.21 | 7,564.6 | — | — | 18,638.96 | |
| BaselineFramework=BOLT, Fine-tuned=false, Network Setting=WAN, Metric Type=Inference Time (s)2026.03 | 5,368.23 | — | — | 5,368.23 | 13,475.92 | 5,368.23 | 5,368.23 | 2,377.48 | 5,368.23 | — | — | 13,475.92 | |
| BaselineFramework=BOLT, Fine-tuned=false, Network Setting=LAN, Metric Type=Inference Time (s)2026.03 | 3,225.6 | — | — | 3,225.6 | 8,464.14 | 3,225.6 | 3,225.6 | 1,370.46 | 3,225.6 | — | — | 8,464.14 | |
| SecDTDFramework=BOLT, Fine-tuned=false, Network Setting=MOBILE, Metric Type=Inference Time (s)2026.03 | 3,172.64 | — | — | 2,150.96 | 5,346.08 | 2,122.99 | 2,183.75 | 1,250.05 | 2,149.83 | — | — | 4,301.92 | |
| SecDTDFramework=BOLT, Fine-tuned=false, Network Setting=WAN, Metric Type=Inference Time (s)2026.03 | 2,241.65 | — | — | 1,519.39 | 3,837.3 | 1,509.03 | 1,549.26 | 887.52 | 1,533.24 | — | — | 3,088.78 | |
| SecDTDFramework=BOLT, Fine-tuned=false, Network Setting=LAN, Metric Type=Inference Time (s)2026.03 | 1,316.63 | — | — | 886.03 | 2,355.26 | 893.78 | 913.45 | 518.37 | 916.02 | — | — | 1,891.44 | |
| Full FTBackbone=RoBERTa-Large, # Params=355.0M2025.09 | 94.7 | — | — | 92.2 | 86.6 | 96.4 | 90.9 | 68 | 92.4 | 88.9 | — | 90.2 | |
| LoRABackbone=RoBERTa-Large, # Params=0.8M2025.09 | 94.7 | — | — | 91.5 | 83.9 | 96 | 89.5 | 65.5 | 90.7 | 87.8 | — | 90.4 | |
| DoRABackbone=RoBERTa-Large, # Params=0.8M2025.09 | 94.6 | — | — | 91.6 | 83.5 | 96 | 89.3 | 65.8 | 91 | 87.8 | — | 90.4 | |
| LoRAr=1Backbone=RoBERTa-Large, # Params=0.1M2025.09 | 94.1 | — | — | 90.2 | 77.9 | 96 | 85.6 | 62 | 84.1 | 85 | — | 89.8 | |
| VeRABackbone=RoBERTa-Large, # Params=0.06M | 0.5M2025.09 | 94.1 | — | — | 89.4 | 79.3 | 95.8 | 89.4 | 65.3 | 89.5 | 86.5 | — | 89.2 | |
| LayerDropLarge#Bits (W-A)=32-32, Size (MB)=1356.1, GFLOPs=23.82024.12 | 93.9 | 89.7 | 89.6 | 88.5 | 86.6 | 95.5 | 91.2 | 66.6 | 92.6 | 88.2 | — | — | |
| HyperAdaptBackbone=RoBERTa-Large, # Params=0.1M2025.09 | 93.8 | — | — | 90.3 | 80.8 | 96.2 | 89.8 | 64.9 | 90.2 | 86.9 | — | 89.3 | |
| BE3R_Electra#Bits (W-A)=32-32, Size (MB)=12782024.12 | 93.7 | 89.6 | 89.3 | 89.6 | 86.3 | 96.8 | 91.5 | 68.3 | 92.3 | 88.6 | — | — | |
| ElasticBERTLarge#Bits (W-A)=32-32, Size (MB)=1279.7, GFLOPs=23.42024.12 | 93.6 | 88 | 88.5 | 90.9 | 83.1 | 95.3 | 92 | 66.3 | 91.7 | 87.7 | — | — | |
| ULR-BERT_LARGEImplementation Source=Our implementation, Batch size=8, 16, 32, 64, Length=128, Epoch=3, lr=3e-52021.05 | 93 | 86.7 | 86 | 72.3 | 71 | 95 | 90.2 | 61.8 | 88.2 | 82.7 | — | — | |
| BERT_LARGEImplementation Source=Our implementation, Batch size=8, 16, 32, 64, Length=128, Epoch=3, lr=3e-52021.05 | 92.8 | 86.1 | 85.6 | 72.1 | 68.8 | 94.9 | 89.6 | 60.5 | 87.3 | 82 | — | — | |
| MLM-BERT_LARGEImplementation Source=Our implementation, Batch size=8, 16, 32, 64, Length=128, Epoch=3, lr=3e-52021.05 | 92.8 | 86.6 | 85.6 | 72.3 | 67.1 | 94.5 | 88.9 | 62.6 | 87.2 | 82 | — | — | |
| BERT_LARGEImplementation Source=In literature2021.05 | 92.7 | 86.7 | 85.9 | 72.1 | 70.1 | 94.9 | 89.3 | 60.5 | 87.6 | 82.2 | — | — | |
| BERT LARGEFinetuning protocol=MNLI-based finetuning for RTE, STS, MRPC2021.04 | 92.6 | 86 | 85.2 | 72 | 78.3 | 94.5 | 89.9 | 60.9 | 87.5 | 83 | — | — | |
| RoBERTa BASEFinetuning protocol=MNLI-based finetuning for RTE, STS, MRPC2021.04 | 92.4 | 87 | 86.5 | 72.5 | 79.6 | 95.8 | 89.7 | 58.8 | 88.3 | 83.4 | — | — | |
| LayerDropBase#Bits (W-A)=32-32, Size (MB)=477.5, GFLOPs=6.72024.12 | 92.2 | 86.4 | 86.5 | 89.9 | 71.1 | 93.7 | 91.6 | 64.5 | 88.6 | 84.9 | — | — | |
| ElasticBERTBase#Bits (W-A)=32-32, Size (MB)=416.4, GFLOPs=6.62024.12 | 92 | 85.3 | 85.9 | 90.2 | 76.5 | 94.3 | 91 | 64.3 | 90.7 | 85.6 | — | — | |
| BE3R_BERT#Bits (W-A)=32-32, Size (MB)=12972024.12 | 91.9 | 85.6 | 85.9 | 88.1 | 71.5 | 93.2 | 90.3 | 59.8 | 89.4 | 84 | — | — | |
| PABEE#Bits (W-A)=32-32, Size (MB)=46, GFLOPs=4.42024.12 | 91.8 | 85.1 | 85.1 | 89.6 | 80.1 | 93 | 90 | 61.2 | 90.1 | 85.1 | — | — | |
| BaselineFramework=BOLT, Fine-tuned=false, Metric Type=Accuracy2026.03 | 91.25 | — | — | 87.68 | 68.23 | 91.74 | 90.53 | 58.8 | 87.97 | — | — | 84.69 | |
| ULR-BERT_BASEImplementation Source=Our implementation, Batch size=8, 16, 32, 64, Length=128, Epoch=3, lr=3e-52021.05 | 91 | 84.6 | 84 | 71.6 | 68 | 94.3 | 89 | 56.5 | 86.6 | 80.6 | — | — | |
| E-PMQModel=FLAN-T5, Merging Algorithm=TIES-Merging, Precision=4-bit2026.05 | 90.81 | — | — | 84.3 | 80.51 | 93 | 81.37 | 69.51 | 85.84 | 83.48 | — | 82.48 | |
| ElasticBERTBase-6L#Bits (W-A)=32-32, Size (MB)=255.9, GFLOPs=3.32024.12 | 90.8 | 84.3 | 84.2 | 89.7 | 74 | 92.7 | 89.7 | 53.7 | 90.2 | 83.3 | — | — | |
| ElasticBERTLarge-6L#Bits (W-A)=32-32, Size (MB)=412.6, GFLOPs=5.92024.12 | 90.8 | 83.5 | 84.3 | 90.1 | 71.1 | 91.9 | 89.6 | 53.9 | 90.1 | 82.8 | — | — | |
| 24hBERTFinetuning protocol=MNLI-based finetuning for RTE, STS, MRPC2021.04 | 90.6 | 84.4 | 83.8 | 70.7 | 75.3 | 93 | 88.5 | 57.1 | 86.8 | 81.1 | — | — | |
| BERT BASEFinetuning protocol=MNLI-based finetuning for RTE, STS, MRPC2021.04 | 90.6 | 84.6 | 84 | 72 | 76.5 | 92.8 | 89.9 | 55.1 | 87.7 | 81.5 | — | — | |
| BERT_BASEImplementation Source=Our implementation, Batch size=8, 16, 32, 64, Length=128, Epoch=3, lr=3e-52021.05 | 90.6 | 84.5 | 83.7 | 71.6 | 67.1 | 92.1 | 87.5 | 53.5 | 85.3 | 79.5 | — | — | |
| BERT_BASEImplementation Source=In literature2021.05 | 90.5 | 84.6 | 83.4 | 71.2 | 66.4 | 93.5 | 88.9 | 52.1 | 87.1 | 79.7 | — | — | |
| SecDTDFramework=BOLT, Fine-tuned=false, Metric Type=Accuracy2026.03 | 90.48 | — | — | 86.75 | 67.87 | 92.78 | 90.57 | 58.06 | 87.87 | — | — | 83.76 | |
| E-PMQModel=FLAN-T5, Merging Algorithm=Task Arithmetic, Precision=4-bit2026.05 | 90.43 | — | — | 84.34 | 82.67 | 92.78 | 79.41 | 69.8 | 84.8 | 83.34 | — | 82.5 | |
| MLM-BERT_BASEImplementation Source=Our implementation, Batch size=8, 16, 32, 64, Length=128, Epoch=3, lr=3e-52021.05 | 90.4 | 84.5 | 83.9 | 71.6 | 66.6 | 94 | 88.1 | 51.9 | 86.2 | 79.7 | — | — | |
| Task ArithmeticModel=FLAN-T5, Merging Algorithm=Task Arithmetic, Precision=Full-precision2026.05 | 90.26 | — | — | 83.61 | 80.51 | 92.2 | 78.43 | 69.8 | 77.82 | 78.79 | — | 57.66 | |
| TIES-MergingModel=FLAN-T5, Merging Algorithm=TIES-Merging, Precision=Full-precision2026.05 | 90.24 | — | — | 83.53 | 81.59 | 91.86 | 78.68 | 70.37 | 78.58 | 79.98 | — | 65.02 | |
| Weight AverageBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 90.08 | — | — | 83.79 | 80.51 | 91.17 | 78.92 | 69.7 | 72 | 78.23 | — | 59.66 | |
| RTNModel=FLAN-T5, Merging Algorithm=Task Arithmetic, Precision=4-bit2026.05 | 90.04 | — | — | 83.78 | 81.59 | 91.17 | 78.92 | 70.18 | 76.1 | 78.51 | — | 56.29 | |
| TSV MergingBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 90.04 | — | — | 83.62 | 79.06 | 92.55 | 80.39 | 69.32 | 82.55 | 81.83 | — | 77.09 | |
| GPTQModel=FLAN-T5, Merging Algorithm=TIES-Merging, Precision=4-bit2026.05 | 89.95 | — | — | 83.2 | 80.51 | 91.28 | 78.19 | 69.8 | 78.45 | 79.66 | — | 65.87 | |
| RTNModel=FLAN-T5, Merging Algorithm=TIES-Merging, Precision=4-bit2026.05 | 89.91 | — | — | 83.56 | 82.67 | 91.06 | 79.66 | 70.37 | 75.42 | 79.53 | — | 63.57 | |
| AWQModel=FLAN-T5, Merging Algorithm=TIES-Merging, Precision=4-bit2026.05 | 89.91 | — | — | 82.81 | 79.78 | 90.83 | 78.43 | 68.55 | 76.51 | 79.37 | — | 68.16 | |
| GPTQModel=FLAN-T5, Merging Algorithm=Task Arithmetic, Precision=4-bit2026.05 | 89.9 | — | — | 83.73 | 79.42 | 92.2 | 77.7 | 69.42 | 77.74 | 78.26 | — | 55.99 | |
| AWQModel=FLAN-T5, Merging Algorithm=Task Arithmetic, Precision=4-bit2026.05 | 89.84 | — | — | 82.91 | 80.51 | 91.06 | 78.43 | 69.51 | 76.85 | 78.8 | — | 61.3 | |
| LayerDropBase-6L#Bits (W-A)=32-32, Size (MB)=313.2, GFLOPs=3.42024.12 | 89.8 | 83.8 | 83.8 | 89.4 | 64.3 | 91.3 | 87.6 | 53.7 | 88.1 | 81.3 | — | — | |
| SWUDIBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 89.8 | — | — | 83.46 | 80.87 | 93 | 77.94 | 69.22 | 85.33 | 82.7 | — | 82 | |
| Task ArithmeticBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 89.79 | — | — | 83.67 | 79.06 | 91.51 | 78.68 | 68.84 | 72.38 | 77.39 | — | 55.18 | |
| SWUDI-ABackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 89.77 | — | — | 83.26 | 80.87 | 92.55 | 83.33 | 68.94 | 85.1 | 82.98 | — | 80 | |
| TA w/ DAREBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 89.73 | — | — | 83.71 | 79.06 | 91.51 | 78.92 | 68.94 | 72.58 | 77.44 | — | 55.1 | |
| OptMergeBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 89.58 | — | — | 83.28 | 79.06 | 93 | 80.39 | 68.36 | 84.27 | 81.03 | — | 70.29 | |
| TIES MergingBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 89.31 | — | — | 83.43 | 79.78 | 91.51 | 78.92 | 68.17 | 74.22 | 76.79 | — | 48.96 | |
| BEExformer (Proposed)#Bits (W-A)=1-1, Size (MB)=14.26, GFLOPs=0.122024.12 | 89.18 | 83.71 | 84.29 | 88.09 | 71.11 | 92.32 | 86.47 | 60.3 | 86.6 | 82.45 | — | — | |
| Iso-CBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 88.63 | — | — | 82.66 | 80.14 | 91.28 | 76.72 | 69.13 | 63.32 | 76.15 | — | 57.35 | |
| LayerDropLarge-6L#Bits (W-A)=32-32, Size (MB)=492.8, GFLOPs=6.02024.12 | 87.1 | 81.4 | 81 | 88.5 | 53.1 | 91.4 | 79.8 | 44.3 | 83 | 76.6 | — | — | |
| FedPowerPrivacy budget=Non-private2026.05 | 86.78 | 95.33 | 92.12 | 86.89 | — | 86.19 | — | — | — | 89.46 | — | — | |
| FFA-LoRAPrivacy budget=Non-private2026.05 | 86.58 | 95.07 | 90.52 | 86.35 | — | 85.25 | — | — | — | 88.75 | — | — | |
| HeadPruneBERTBase#Bits (W-A)=32-32, Size (MB)=332.3, GFLOPs=4.72024.12 | 86.1 | 71 | 79.7 | 84.7 | 62.5 | 89.4 | 80.2 | 48.7 | 85.2 | 76.4 | — | — | |
| FedLoRAPrivacy budget=Non-private2026.05 | 85.69 | 94.95 | 89.6 | 85.48 | — | 84.21 | — | — | — | 87.99 | — | — | |
| WUDI MergingBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 84.64 | — | — | 82.7 | 71.48 | 93 | 78.43 | 68.65 | 83.82 | 79.36 | — | 72.18 | |
| TIES w/ DAREBackbone=Flan-T5-base, Tuning=LoRA fine-tuned2026.06 | 84.44 | — | — | 82.23 | 76.9 | 89.56 | 79.9 | 31.16 | 75.94 | 65.07 | — | 0.43 | |
| FedPowerPrivacy budget=epsilon = 92026.05 | 84.12 | 94.04 | 89.72 | 83.61 | — | 84.16 | — | — | — | 87.13 | — | — | |
| FedPowerPrivacy budget=epsilon = 62026.05 | 83.72 | 93.72 | 88.2 | 83.12 | — | 84.01 | — | — | — | 86.55 | — | — | |
| FedLoRAPrivacy budget=epsilon = 92026.05 | 83.24 | 93.46 | 87.86 | 82.95 | — | 83.96 | — | — | — | 86.3 | — | — | |
| FFA-LoRAPrivacy budget=epsilon = 92026.05 | 82.9 | 93.69 | 87.61 | 82.19 | — | 81.26 | — | — | — | 85.53 | — | — | |
| FedLoRAPrivacy budget=epsilon = 62026.05 | 82.88 | 93.12 | 86.97 | 82.6 | — | 83.7 | — | — | — | 85.85 | — | — | |
| FFA-LoRAPrivacy budget=epsilon = 62026.05 | 82.75 | 93.23 | 87.52 | 82.21 | — | 81.47 | — | — | — | 85.44 | — | — | |
| FedPowerPrivacy budget=epsilon = 32026.05 | 82.61 | 93.03 | 84.99 | 81.94 | — | 83.3 | — | — | — | 85.18 | — | — | |
| FFA-LoRAPrivacy budget=epsilon = 32026.05 | 82.6 | 92.43 | 87.06 | 81.94 | — | 80.85 | — | — | — | 84.98 | — | — | |
| FedLoRAPrivacy budget=epsilon = 32026.05 | 81.7 | 91.28 | 83.89 | 81.26 | — | 79.56 | — | — | — | 83.53 | — | — | |
| SecDTDFramework=BOLT, Fine-tuned=false, Network Setting=LAN, Metric Type=Speedup2026.03 | 2.45 | — | — | 3.64 | 3.59 | 3.61 | 3.53 | 2.64 | 3.52 | — | — | 4.47 | |
| SecDTDFramework=BOLT, Fine-tuned=false, Network Setting=WAN, Metric Type=Speedup2026.03 | 2.39 | — | — | 3.53 | 3.51 | 3.56 | 3.47 | 2.68 | 3.5 | — | — | 4.36 | |
| SecDTDFramework=BOLT, Fine-tuned=false, Network Setting=MOBILE, Metric Type=Speedup2026.03 | 2.38 | — | — | 3.52 | 3.49 | 3.56 | 3.46 | 2.71 | 3.52 | — | — | 4.33 |