Natural Language Understanding on GLUE (8-Task Subset: CoLA, SST-2, MRPC, QQP, MNLI, QNLI, RTE, STS-B)
66.7CoLA AccuracyMP-ISMoE† (T5-large)
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| MP-ISMoE† (T5-large)Backbone=T5-large, Params (%)=1.75, Memory Train (G)=9.9, Memory Test (G)=2.822026.04 | 66.7 | 96.5 | 93.1 | 90 | 88.7 | 94.7 | 79.8 | 92.4 | 87.7 | |
| MP-ISMoE‡ (T5-large)Backbone=T5-large, Params (%)=0.81, Memory Train (G)=7.6, Memory Test (G)=2.802026.04 | 66.4 | 96.5 | 93.4 | 90.3 | 88.9 | 94.9 | 81.6 | 92.7 | 88.1 | |
| UniPT (T5-large)Backbone=T5-large, Params (%)=0.92, Memory Train (G)=9.1, Memory Test (G)=2.822026.04 | 65.7 | 95.8 | 92 | 89.7 | 88.2 | 94.2 | 79.6 | 92 | 87.2 | |
| SHERL (T5-large)Backbone=T5-large, Params (%)=0.64, Memory Train (G)=7.1, Memory Test (G)=2.802026.04 | 65.6 | 95.8 | 92.9 | 89.6 | 88.6 | 94.2 | 80.8 | 92.1 | 87.5 | |
| LST (T5-large)Backbone=T5-large, Params (%)=1.23, Memory Train (G)=12.2, Memory Test (G)=2.882026.04 | 65.3 | 95.7 | 91.6 | 89.7 | 88.6 | 94.1 | 79.9 | 92.4 | 87.1 | |
| AdapterBackbone=T5-base, Params (%)=1.63, Memory Train (G)=13.0, Memory Test (G)=0.872026.04 | 64.4 | 94.2 | 88.9 | 88.9 | 86.4 | 93.1 | 75.1 | 91.1 | 85.3 | |
| MP-ISMoE†Backbone=T5-base, Params (%)=2.41, Memory Train (G)=3.2, Memory Test (G)=0.862026.04 | 63.4 | 94.6 | 91.6 | 89.1 | 85.7 | 93.4 | 70.9 | 90 | 84.8 | |
| LoRABackbone=T5-base, Params (%)=1.71, Memory Train (G)=12.6, Memory Test (G)=0.862026.04 | 63.3 | 94.3 | 90.1 | 89 | 86.3 | 93.2 | 75.5 | 90.9 | 85.3 | |
| Fully-FTBackbone=T5-base, Params (%)=100, Memory Train (G)=17.6, Memory Test (G)=0.862026.04 | 62.8 | 93.9 | 91.9 | 89.9 | 86.2 | 92.5 | 74.1 | 90.3 | 85.2 | |
| MP-ISMoE‡Backbone=T5-base, Params (%)=1.48, Memory Train (G)=3.3, Memory Test (G)=0.872026.04 | 62.4 | 94.3 | 89.6 | 89.3 | 85.9 | 93.2 | 72.8 | 91.5 | 84.9 | |
| UniPTBackbone=T5-base, Params (%)=1.36, Memory Train (G)=2.9, Memory Test (G)=0.862026.04 | 62.2 | 94.2 | 90.8 | 88.9 | 85.5 | 93.3 | 69.8 | 89.7 | 84.3 | |
| BitFitBackbone=T5-base, Params (%)=0.13, Memory Train (G)=10.7, Memory Test (G)=0.862026.04 | 61.8 | 94.3 | 91 | 88.7 | 85.6 | 93.1 | 67.6 | 90.8 | 84.1 | |
| SHERLBackbone=T5-base, Params (%)=0.85, Memory Train (G)=2.9, Memory Test (G)=0.872026.04 | 61.1 | 93.7 | 89.4 | 88.8 | 85.3 | 93.3 | 71.9 | 90.9 | 84.3 | |
| LSTBackbone=T5-base, Params (%)=1.74, Memory Train (G)=5.5, Memory Test (G)=0.882026.04 | 58.1 | 94.1 | 90.4 | 88.8 | 85.6 | 93.3 | 71.9 | 90.7 | 84.1 | |
| PromptBackbone=T5-base, Params (%)=0.03, Memory Train (G)=22.2, Memory Test (G)=0.872026.04 | 0 | 90.3 | 74.6 | 88.5 | 82.5 | 92.5 | 59.5 | 90.1 | 72.2 |