Natural Language Understanding on GLUE (7-Task Subset: CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2)
91.91MRPC ScoreFourierMoE
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FourierMoEBackbone=RoBERTa-large, # Params (%)=0.422026.04 | 91.91 | 90.33 | 91.52 | 96.33 | 89.53 | 85.52 | 94.66 | 91.4 | — | — | — | — | — | |
| MTL-LoRABackbone=Qwen3-8B, Trainable Parameters (%)=0.202026.04 | 89.46 | 91.93 | 91.39 | 96.67 | 92.77 | 66.32 | 95.77 | 89.18 | — | — | — | — | — | |
| SAMoRA w/o RouterBackbone=Qwen3-8B, Trainable Parameters (%)=0.202026.04 | 89.46 | 92.08 | 91.41 | 97.13 | 91.33 | 68.19 | 95.91 | 89.36 | — | — | — | — | — | |
| SAMoRABackbone=Qwen3-8B, Trainable Parameters (%)=0.182026.04 | 89.21 | 91.96 | 91.41 | 97.13 | 94.22 | 69.75 | 96.17 | 89.98 | — | — | — | — | — | |
| SAMoRA w/o ScalingBackbone=Qwen3-8B, Trainable Parameters (%)=0.182026.04 | 88.97 | 91.93 | 90.84 | 96.9 | 91.33 | 66.43 | 95.93 | 88.9 | — | — | — | — | — | |
| LoRABackbone=Qwen3-8B, Trainable Parameters (%)=0.212026.04 | 88.2 | 91.84 | 91.12 | 96.5 | 91.16 | 64.06 | 96.01 | 88.41 | — | — | — | — | — | |
| SAMoRA w/o LorthBackbone=Qwen3-8B, Trainable Parameters (%)=0.152026.04 | 87.99 | 91.99 | 90.89 | 97.01 | 90.61 | 68.32 | 96.11 | 88.99 | — | — | — | — | — | |
| MoLoRABackbone=RoBERTa-large, # Params (%)=4.502026.04 | 87.75 | 89.36 | 91.45 | 96.1 | 84.11 | 83.94 | 93.9 | 89.52 | — | — | — | — | — | |
| SAMoRA w/o LmatchBackbone=Qwen3-8B, Trainable Parameters (%)=0.152026.04 | 87.25 | 91.88 | 90.63 | 97.01 | 91.69 | 68.73 | 95.97 | 89.02 | — | — | — | — | — | |
| FourierFTBackbone=RoBERTa-large, # Params (%)=0.422026.04 | 86.27 | 87.19 | 85.48 | 95.2 | 83.39 | 78.01 | 91.59 | 86.73 | — | — | — | — | — | |
| AdaMoLEBackbone=RoBERTa-large, # Params (%)=4.562026.04 | 86.03 | 89.21 | 91.48 | 95.76 | 83.75 | 83.99 | 93.64 | 89.12 | — | — | — | — | — | |
| MiLoRABackbone=RoBERTa-large, # Params (%)=4.002026.04 | 86.02 | 89.51 | 91.33 | 96.1 | 84.83 | 84.65 | 94.12 | 89.51 | — | — | — | — | — | |
| rsLoRABackbone=RoBERTa-large, # Params (%)=4.002026.04 | 86.02 | 88.97 | 90.75 | 95.98 | 84.12 | 83.51 | 93.84 | 89.03 | — | — | — | — | — | |
| FFTBackbone=RoBERTa-large, # Params (%)=1002026.04 | 85.29 | 89.83 | 91.58 | 95.98 | 84.84 | 84.27 | 94.49 | 89.47 | — | — | — | — | — | |
| FFT MoEBackbone=RoBERTa-large, # Params (%)=6982026.04 | 85.05 | 90.2 | 92.2 | 96.22 | 84.48 | 86.02 | 95.1 | 89.9 | — | — | — | — | — | |
| HydraLoRABackbone=RoBERTa-large, # Params (%)=2.752026.04 | 85.04 | 89.34 | 91.02 | 95.52 | 81.22 | 83.89 | 93.87 | 88.56 | — | — | — | — | — | |
| HydraLoRABackbone=Qwen3-8B, Trainable Parameters (%)=0.202026.04 | 85.04 | 91.9 | 90.68 | 96.55 | 75.45 | 67.04 | 90.51 | 85.31 | — | — | — | — | — | |
| GOATBackbone=RoBERTa-large, # Params (%)=4.502026.04 | 84.55 | 89.55 | 91.4 | 96.21 | 85.56 | 86.86 | 94.19 | 89.76 | — | — | — | — | — | |
| DoRABackbone=RoBERTa-large, # Params (%)=4.002026.04 | 84.07 | 89.52 | 91.24 | 95.99 | 84.48 | 85.33 | 93.54 | 89.17 | — | — | — | — | — | |
| MoeLoRABackbone=Qwen3-8B, Trainable Parameters (%)=0.602026.04 | 83.82 | 91.71 | 90.07 | 96.67 | 91.69 | 67.01 | 95.88 | 88.12 | — | — | — | — | — | |
| LoRABackbone=RoBERTa-large, # Params (%)=4.002026.04 | 83.33 | 89 | 90.06 | 95.64 | 84.47 | 83.41 | 93.28 | 88.46 | — | — | — | — | — | |
| PiSSABackbone=RoBERTa-large, # Params (%)=4.002026.04 | 82.84 | 88.94 | 91.24 | 95.98 | 73.29 | 69.12 | 93.59 | 85 | — | — | — | — | — | |
| MultiLoRABackbone=Qwen3-8B, Trainable Parameters (%)=0.672026.04 | 80.88 | 90.83 | 89.57 | 97.02 | 85.56 | 58.5 | 94.12 | 85.21 | — | — | — | — | — | |
| SaES-ID-CRModel=OPT-6.7B2026.03 | 80.18 | 33.54 | 54.77 | 86.17 | 57.76 | 8.4 | 58.46 | — | — | — | — | — | — | |
| (1+1)-ES-ID-CRModel=OPT-6.7B2026.03 | 74 | 30.7 | 48.02 | 71.88 | 54.95 | 11.42 | 56.3 | — | — | — | — | — | — | |
| TF (Sin.)Architecture=TF (Sin.)2026.05 | 67.5 | — | 75.8 | 81.1 | — | — | 56.9 | — | 12.4 | 59.1 | 70.7 | 49 | 50.5 | |
| TF (NoPE)Architecture=TF (NoPE)2026.05 | 64 | — | 77.8 | 79.1 | — | — | 58.8 | — | 9.7 | 59 | 72.8 | 50.5 | 51.1 | |
| MLP-LDRUArchitecture=MLP-LDRU2026.05 | 63.2 | — | 79.1 | 80.9 | — | — | 56.1 | — | 8.5 | 57 | 76.9 | 58.3 | 57.9 | |
| TF (ALiBi)Architecture=TF (ALiBi)2026.05 | 62.6 | — | 78.3 | 79.3 | — | — | 58.1 | — | 9.6 | 56.2 | 74 | 53.9 | 52.8 | |
| BBTModel=OPT-6.7B2026.03 | 57.33 | 32.92 | 50.41 | 81.22 | 56.53 | 10.3 | 58.22 | — | — | — | — | — | — |